%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
1. Introduction
Consider a Bayesian agent assigning degrees of belief to various propositions. There are two well-known concepts in this context that, I shall demonstrate, can be fruitfully connected: scoring rules and confirmation measures. Scoring rules (see, e.g., Winkler Reference Winkler1969; Bickel Reference Bickel2007; Gneiting and Raftery Reference Gneiting and Raftery2007) can play at least three different roles: (i) to force agents to make their degrees of belief follow the rules of probability calculus, (ii) to force their degrees of belief in given hypotheses to equal the probabilities they assign to those hypotheses, and (iii) to quantify the inaccuracy of their degrees of belief.
Irrespective of its use, a scoring rule associates with each proposition H (chosen from a partition of hypotheses over some domain) a penalty, the amount of which depends on the agent’s degree of belief (or credence) b in that proposition and on the proposition’s truth value t (with 
for true and 
for false). For an illustrative example, consider the “quadratic scoring rule”—chosen because it is the simplest, although not necessarily the best—which assigns a penalty
Suppose our agent assigns a probability p to the truth of proposition H. For that agent, the expected value of the penalty incurred on that hypothesis is then
The second line shows that the expected penalty is minimized by choosing the degree of belief b equal to the probability p. This property makes the scoring rule proper. Furthermore, the penalty can be minimized only by choosing 
, and that makes the quadratic scoring rule a strictly proper one. And thus agents have, to the extent that they care about minimizing expected penalties, an incentive to align degrees of belief with probabilities. This demonstrates use and interpretation ii of the scoring rule.
One can view the above argument as a Dutch Book argument, in that an agent who is betting on his or her beliefs can be made to lose money if those degrees of belief do not equal probabilities (see De Finetti Reference de Finetti1990; Schervish, Kadane, and Seidenfeld Reference Schervish, Kadane, Seidenfeld and Bernardo2003). The same argument, using the same scoring rule, can be interpreted differently (Joyce Reference Joyce1998; see also Leitgeb and Pettigrew Reference Leitgeb and Pettigrew2010; Levinstein Reference Levinstein2012): one can view the expected penalty as a measure of the inaccuracy of the agent’s belief (interpretation iii). With this interpretation, the agent must align credences with probabilities in order to be the least inaccurate. The philosophical advantage of interpreting the scoring rule this way is that with it the agent has an epistemic rather than a pragmatic reason for adhering to certain beliefs. I will stick with this interpretation in the following, and I will assume that the agent chooses 
for that epistemic reason. (I simply accept that a Bayesian agent’s credence assignments conform to probability calculus, and interpretation i of scoring rules plays no role in this study.)
Suppose our agent learns a new piece of evidence E concerning hypothesis H, which makes the agent update his or her degree of belief from 
to 
(in different notation, 
and 
). In the light of the new probability assigned to H, the agent’s previous degree of belief b is now revealed to be less accurate. Indeed, according to the quadratic scoring rule, the change in inaccuracy thanks to the new evidence is negative and equals
Here the expected value of the penalty incurred on the previous belief b is based on the updated probability p′.
The basic observation of the present study is now this: equation (3) quantifies how much the evidence E has changed the agent’s belief in the hypothesis H, but Bayesian confirmation measures are meant to do exactly this, too. So why not set the degree of confirmation equal to 
?
2. Two Confirmation Measures from Scoring Rules
To answer this question, we first have to overcome a technical obstacle: whereas a scoring rule does not distinguish between evidence being undermining or confirming belief in H, a confirmation measure does make this distinction, namely, by including a sign in the amount of confirmation. That is, if we denote a generic measure of confirmation by C(H; E), then, by convention, 
indicates the agent’s belief in the truth of H is undermined by E, while if 
it is supported. Fortunately, in the case of the quadratic scoring rule, this obstacle is easily overcome by explicitly including this sign in the definition of the degree of confirmation (by setting it equal to either 
or 
). I will in addition show in this section how the same simple solution works just as well for another well-established scoring rule. (I relegate to the appendix the discussion of two further scoring rules for which this simple solution does not work. Although one can still construct confirmation measures connected to these scoring rules, the measures are no longer uniquely determined by the scoring rules and thereby lose some appeal.)
2.1. Quadratic Confirmation Measure
Denoting the confirmation measure based on 
by Q we include a (negative) positive sign for (dis)confirming cases by setting Q equal to
such that
This measure of confirmation, which we could call the “quadratic confirmation measure,” is trivially ordinally equivalent to the difference measure 
(as proposed by Carnap [Reference Carnap1962]). For example, the measures d and Q always agree on (i) which of any two pieces of evidence supports the belief in a given proposition more than the other and (ii) which of any pair of propositions is supported more than the other by a given piece of evidence. The measure Q has at least one advantage over d (apart from being derivable from a scoring rule): the linear dependence on b′ and b leaves d, unlike Q, open to attacks like (one version of) the Popper-Miller argument against inductive inferences (Fitelson Reference Fitelson1999). On the other hand, all positive features of the difference measure discussed in Fitelson (Reference Fitelson1999) and Eells and Fitelson (Reference Eells and Fitelson2002) are possessed by Q as well.
The quadratic scoring rule is not optimal according to the pragmatic criteria put forward in Bickel (Reference Bickel2007), and there is no reason to expect the corresponding confirmation measure Q to be optimal, either. So the next order of business will be to discuss a second scoring rule and see whether and how one can construct a (possibly better) confirmation measure from it.
Before doing this, however, there is one more issue to address, and that is to give the reason for using the difference, in equation (3), between the old and new inaccuracies for quantifying the change in accuracy, rather than, say, their ratio, or the logarithm of their ratio (for confirmation measures that same choice is open for debate). The reason is this. Two scoring rules are considered equivalent if and only if they are related to each other via a positive affine transformation, that is, via multiplication with a positive constant and addition of another constant. Under a positive affine transformation, the difference between old and new penalties changes by just the multiplication factor, and that transformation in turn leaves confirmation measures equivalent. The ratio of old and new penalties, in contrast, changes in quite arbitrary ways and can, for instance, be made infinite (and thereby meaningless) by an appropriate choice of the affine transformation (provided old and new penalties are not equal). Only by taking the difference between old and new penalties do equivalent scoring rules lead to equivalent confirmation measures.
2.2. Logarithmic Confirmation Measure
In Reference Good1952 Good proposed a logarithmic scoring rule of the form
Using the inaccuracy interpretation of this scoring rule, updating an agent’s degree of belief from b to b′ makes that belief less inaccurate. This can be shown by applying the Gibbs inequality (
with equality only for 
) twice, so that
with equality only for 
. This also shows that this rule is strictly proper. By following the same steps as before, we construct the corresponding measure of confirmation, which we denote by L, by setting
such that
This measure (as well as Q) satisfies the Hypothesis Symmetry (Eells and Fitelson Reference Eells and Fitelson2002). That is, under the replacement of a hypothesis H by the catchall −H, the confirmation measure changes sign. In the notation used here, this means that L changes sign under the simultaneous replacements 
and 
. In words, evidence confirms H just as much as it disconfirms −H. Eells and Fitelson regard this as a good symmetry for confirmation measures to possess. On the other hand, Eells and Fitelson also discuss two symmetries that, they argue, confirmation measures should not possess, and it turns out that, indeed, neither L nor Q satisfy Evidence Symmetry (i.e., 
for all evidence E and any hypothesis H) or Commutativity Symmetry (i.e., 
).
I end with two technical observations. First, L increases with b′ for fixed b with 
and decreases with b′ for fixed b when 
, as a confirmation measure indeed should. Second, for relatively small changes in belief, that is, for 
with 
, we have (for not-too-large values of b)
This observation helps one to see that the two measures Q and L are not ordinally equivalent. Just consider two cases of supporting evidence, case 1 where 
and 
, and case 2 where 
and 
. According to measure Q the first case provides less support than does the second, but according to measure L case 1 provides more support.
3. A Test: The Bit String Guessing Game
In order to see whether there is any merit to at least one of the confirmation measures just constructed, I consider a simple game. The goal is to guess a secret M-bit string, which has been picked at random from all 
possible M-bit strings. Corresponding to each possible string k there is the proposition Hk which states that string k is the secret one. Suppose our agent discovers the first piece of evidence, namely, that the first bit is “0.” The agent updates her or his degrees of belief from 
for all N propositions to (i) 
for the propositions about those bit strings that start with “0” and (ii) 
for the remaining ones. Subsequently discovering the next n bits would be described in an analogous way, by just replacing N by N/2n, and so on.
3.1. Irrelevant Conjunction
This simple game provides an instance of a problem that has been dealt with successfully within Bayesian confirmation theory, namely, the problem of irrelevant conjunction (see, e.g., Hawthorne and Fitelson Reference Hawthorne and Fitelson2004). For, given that the evidence concerns only the first bit, each proposition Hk can be viewed as a conjunction of a relevant statement about the first bit with irrelevant statements about the other bits. The problem of irrelevant conjunction is, briefly, this: if evidence E confirms some hypothesis H, in the sense of increasing the probability assigned to the truth of H, then it also confirms, in the same sense, the composite hypothesis H & X, where X is any irrelevant proposition. Intuitively, that does not seem quite right. The way Bayesian confirmation theory has solved this problem is by showing that, at least for some confirmation measures, the degree of confirmation, although still positive, is diminished by adding irrelevant propositions. I will demand the same feature of the measures of confirmation considered here.
3.2. Desiderata
What do the various Bayesian measures of confirmation, both previously known ones and the ones proposed here, have to say about the evidence obtained in the bit-string guessing game? I will judge the confirmation measures on four criteria. I state the criteria in terms of a generic measure of confirmation denoted as C, and I abbreviate 
.
(I) The evidence is literally 1 bit of information. The total amount of confirmation, defined here as the sum
, ought to reflect this fact, in that, taken over all values of M, this sum ought to be bounded from above, and it ought to be bounded from below away from zero. In other words, the total amount of confirmation should never be zero nor infinite, not even in the limit of .(II) A necessary property of C, if it is to handle the problem of irrelevant conjunction properly, is that for any k the absolute value
strictly decrease with increasing number of bits M, unless , when it should stay equal to zero for all M.(III) For those propositions for which the evidence is undermining, it is decisive in that no further evidence can change the agent’s degree of belief. But the confirming evidence is not decisive, unless
. Thus, for , the amount of disconfirmation brought about by the one bit of evidence ought to be larger (in absolute value) than the amount of confirmation.(IV) For
the amount of confirming evidence ought to equal the amount of undermining evidence, as in both cases the evidence is decisive and the situation is symmetric under the interchange .
, ought to reflect this fact, in that, taken over all values of M, this sum ought to be bounded from above, and it ought to be bounded from below away from zero. In other words, the total amount of confirmation should never be zero nor infinite, not even in the limit of 
.
strictly decrease with increasing number of bits M, unless 
, when it should stay equal to zero for all M.
. Thus, for 
, the amount of disconfirmation brought about by the one bit of evidence ought to be larger (in absolute value) than the amount of confirmation.
the amount of confirming evidence ought to equal the amount of undermining evidence, as in both cases the evidence is decisive and the situation is symmetric under the interchange 
.3.3. Test Results
I apply these criteria to five different measures of confirmation, with the following results.
Q. According to the quadratic measure Q we have the following degrees of confirmation:
This measure passes requirements II and IV but fails I and III. It fails I because in the limit 
we have 
.
L. According to the logarithmic measure L derived from Good’s scoring rule, we have
This measure passes all four tests. For 
the degree of confirmation is
, precisely quantifying the one bit of information that has been obtained as evidence (one bit equals 
nats [natural units]).
l. Consider the well-known measure l, the log likelihood ratio (as advocated by Good [Reference Good1983]). Here the evidence is quantified as
This measure passes III and IV but fails I. The confirmation part passes II, but the disconfirmation part does not. (More about how and why l fails here follows below.)
d. The difference measure satisfies
and the total amount of confirmation, 
, equals 1. This measure passes I, II, and IV, but it fails III.
r. The log probability ratio, denoted by r, was advocated as the best Bayesian confirmation measure in Milne (Reference Milne1996) and originates from Keynes (Reference Keynes2004). It yields
This measure passes only III and fails the other three criteria.
3.4. L versus l
Only one measure of the five considered here satisfies all four criteria, namely, the measure L. The reason that the measure l fails some criteria is because it assigns an infinite amount of confirmation to decisive evidence, but there is a good argument for doing so. No finite amount of future evidence is able to reverse the verdict that a degree of belief is zero or that a degree of belief equals unity. One could say, then, that the main difference between measures L and l is, roughly, that the former directly quantifies the present evidence, whereas the latter quantifies the future evidence needed to undo the present evidence.
A similar characterization of the difference between L and l presents itself when examining the following argument (from Fitelson Reference Fitelson2001) given in favor of l. The argument states that decisive evidence for or against a hypothesis H should always count as the maximal amount of evidence, irrespective of the prior probability assigned to H. The latter condition makes this “maximality” requirement incompatible with criterion II. The measure l satisfies this maximality requirement by always assigning an infinite degree of confirmation to decisive evidence. In particular, it does this irrespective of how many bits the evidence is composed of. In contrast, L always assigns a finite amount 
to decisive evidence, with b the prior probability assigned to the hypothesis that is now conclusively confirmed. This implies, for instance, that, for the test case considered here, M decisive bits taken together confer a degree of confirmation of 
according to L, thus precisely capturing the amount of information contained in that evidence.
3.5. Q versus d
Although the measures Q and d are trivially ordinally equivalent, one way in which Q performs better than d was mentioned earlier, in section 2.1. The bit string test, on the other hand, gives an argument in favor of d and against Q: whereas d fails only requirement III, Q fails in addition requirement I.
3.6. L versus Q
The fact that the confirmation measure L derived from Good’s logarithmic scoring rule was found to be superior to the measure Q derived from the quadratic scoring rule happily agrees with the former scoring rule being considered superior to the latter rule on the basis of pragmatic criteria discussed by Bickel (Reference Bickel2007). We may draw the conclusion, also, that as an epistemic measure of inaccuracy, the logarithmic scoring rule is better than the quadratic scoring rule. This conclusion agrees with Levinstein (Reference Levinstein2012) but is contra Leitgeb and Pettigrew (Reference Leitgeb and Pettigrew2010).
4. Conclusions
The point of this article was to demonstrate a connection between scoring rules and Bayesian confirmation measures by showing how to construct measures of confirmation from two particular scoring rules. The idea behind this construction is as follows. First, within Bayesian confirmation theory, how much a given piece of evidence supports or undermines a hypothesis is determined by how much the agent’s belief in that hypothesis changed. Second, this change in belief may be quantified by how much less inaccurate that belief has become. Third, the decrease in inaccuracy is appropriately measured by a scoring rule (in Joyce’s [Reference Joyce1998] interpretation).
One surprise—given the plethora of existing measures of confirmation—is that one of these measures is genuinely new. Another surprise is that in a simple test that new measure (based on a scoring rule proposed by Good in Reference Good1952) emerges as the best, even when compared to established confirmation measures. In standard notation its magnitude is defined as
This measure shares several good properties with the log likelihood ratio (also connected to Good’s name), which in the same notation can be written so as to bring out the similarities between the two, as
I argued that l can be seen as quantifying the future evidence needed to undo evidence E, whereas L directly quantifies the information contained in E.
The above legitimizes L as a valid Bayesian confirmation measure. Further tests in other relevant contexts are an obvious avenue of research.
Appendix
This appendix will show how two other scoring rules can be used to construct two confirmation measures in less straightforward ways. The measures obtained here are by no means unique. Such measures may have their uses, but given the strength of the measure L, the results here are presented merely for the sake of completeness. (Neither measure satisfies all criteria of sec. 3.2.)
Asymmetric Logarithmic Rule
In the case of the two scoring rules discussed in the main text, the decrease of inaccuracy only depends on the old and new degrees of belief in the proposition under consideration. In particular, it does not depend on any other degrees of belief. For other scoring rules this is not necessarily the case. In order to treat the role of other propositions in those scoring rules, we suppose, for simplicity and convenience, that we can create a countable partition of our domain of interest. That is, we assume we have a set of N mutually exclusive propositions 
such that exactly one Hi must be true (we may take the limit of 
at the end of a calculation). Moreover, it will be advantageous to also assume that all propositions are simple, in that the probability of any evidence to occur is fully determined by specifying which of the propositions in the partition is true. We denote our agent’s old and new degrees of belief in proposition Hi by bi and 
, respectively. We have, then, 
.
One may define an asymmetric version of Good’s rule (see Bickel [Reference Bickel2007] or Gneiting and Raftery [Reference Gneiting and Raftery2007]) by setting the penalty as
So in this case a penalty is incurred only for the proposition H that turns out to be true. The expected change in inaccuracy upon changing the degree of belief from b to b′ is now
The inaccuracy may actually increase, namely, when 
. At first sight this may seem to automatically overcome the technical obstacle mentioned above, in that it may seem that we could just define a confirmation measure by setting 
. However, this fails because the amount of disconfirmation would equal zero whenever 
, whereas it should be strictly negative.
The scoring rule (A1) is, in spite of appearances perhaps, strictly proper, when one takes into account that, when 
, at least one of the other degrees of belief must increase in such a way that the total expected change in inaccuracy actually always does decrease. We can see this by examining the following auxiliary quantity:
By the Gibbs inequality this quantity is never positive and equal to zero only if 
. Moreover, we have
This sum represents the change in the sum of the inaccuracies of our agent’s beliefs about all the propositions in the partition (cf. eq. [A2]). This sum will be referred to as “the total inaccuracy.” (It in fact equals the Kullback-Leibler divergence between the old and the new probability distributions, which has the interpretation as the amount of information lost when the old probability distribution is used instead of the updated one.) The change in total inaccuracy is always negative unless none of the agent’s credences change.
After these preliminary remarks, we are ready to define a measure of confirmation as
We might denote A as the “asymmetric logarithmic measure.” Even though we had to invoke other propositions to demonstrate the strict properness of the scoring rule, A depends only on the old and new credences in the proposition under consideration. This measure is constructed so as to satisfy
This constraint is (much) weaker than (5) or (9), as it restricts only the sum, and not each individual term. As such, the choice of measure A is not unique and one might even say it is ad hoc. The measure satisfies criteria I, II, and III, but not IV. It does not satisfy the Hypothesis Symmetry of Eells and Fitelson (Reference Eells and Fitelson2002).
Spherical Scoring Rule
The spherical scoring rule depends explicitly on the distribution of values {bj} through the Euclidean length of the vector
For proposition Hi the penalty is set as
Again, a penalty is incurred only if proposition Hi turns out to be true, and the penalty increases from 0 for 
to a maximum of 1 for 
. Upon changing beliefs, the change in expected value of the penalty incurred on proposition Hi is
Analogously to the case of the asymmetric logarithmic scoring rule, 
may be positive for some values of i, but the total inaccuracy never increases. The latter statement follows from the Cauchy-Schwarz inequality
with equality only for 
. Now we can rewrite the change in total inaccuracy as
The right-hand side is a sum of all nonpositive terms, but the conditions under which the individual terms in the sum are zero are the wrong conditions for confirmation measures, namely, 
, instead of what would be the correct conditions 
. To fix this problem, let us define for 
the following positive “global” quantity, which depends on the whole probability distribution,
where the subscript E indicates that the value of 
depends explicitly on 
and thereby on the evidence E. Then we can rewrite
as a sum of nonpositive terms that vanish only when 
. This enables us to define
so as to satisfy the constraint
The two measures Q and S are obviously similar, but they are not ordinally equivalent. While they always agree on which of any pair of propositions is more confirmed by a given piece of evidence, they do not necessarily agree on which of two pieces of evidence provides more confirmation for a given proposition, thanks to the dependence of 
on E.
Finally, the measure S fails criteria I and III and does not satisfy Eells and Fitelson’s Hypothesis Symmetry.
