1. Introduction
Bayesian updating, also known as ‘conditionalization’, is a rule specifying how a prior probability distribution should be updated to a posterior distribution in the light of new information. The rule is often interpreted epistemically: it specifies how an agent should change its epistemic state over time in response to new evidence, where an ‘epistemic state’ is represented by a probability distribution over some specified set of alternatives.
There has been considerable philosophical discussion of the normative status of Bayesian updating. Does rationality require an agent to update using the Bayesian rule? If so, can the rule be justified from more primitive rationality requirements? Is it the unique rule that can be so justified? Are there situations in which Bayesian updating should not be used? Works addressing these questions include Brown (Reference Brown1976), van Fraassen (Reference van Fraassen1989), Christensen (Reference Christensen1991), Maher (Reference Maher1992), Lewis (Reference Lewis1999), Greaves and Wallace (Reference Greaves and Wallace2006), and Leitgeb and Pettigrew (Reference Leitgeb and Pettigrew2010).
Here I focus on a different although related issue. Instead of asking whether Bayesian updating is a requirement of rationality, I ask whether it is a requirement of evolutionary optimality. Changing one’s beliefs in response to evidence is an aspect of cognition, and it seems likely that at least some aspects of cognition, animal and human, have been shaped by natural selection. This prompts the question of how selection would have operated on updating rules. An organism’s updating rule affects its epistemic state, which in turn affects its behavior, so different updating rules will lead to different behaviors and, thus, have different consequences for survival and reproduction. Can we show that organisms using Bayesian updating would have enjoyed a selective advantage over those updating in some other way?
This question is an aspect of the broader issue of whether natural selection will tend to produce rational behavior. A number of authors have examined whether rational-choice norms such as transitivity of preference, maximization of expected utility, and consistency of time preference can be derived from an underlying evolutionary model (Skyrms Reference Skyrms1995; Robson Reference Robson1996; Cooper Reference Cooper2003; Houston, McNamara, and Steer Reference Houston, McNamara and Steer2007). However, the question of whether Bayesian updating can be derived from evolutionary principles has never been considered, so far as I know. Here I sketch a tentative answer to that question.
2. Bayesianism in Evolutionary Biology
Some readers may be surprised by our discussing Bayesian principles in an evolutionary context, on the grounds that nonhuman animals lack the cognitive sophistication required to have probabilistic beliefs about the world, less still to update them. But in fact, behavioral ecologists make widespread use of Bayesian ideas to think about animal behavior, especially in foraging theory (e.g., McNamara, Green, and Olsson Reference McNamara, Green and Olsson2006).
A typical Bayesian approach in foraging theory is to assume that animals have some prior information about an environmental parameter, represented by a probability distribution. For example, the animal may ‘know’ that food patches are of two types, good and bad, whose relative frequencies are 1/3 and 2/3 respectively. This information may be either genetically encoded or acquired by the animal through its experience. As the animal begins to forage, it updates its belief about the type of patch it is in. If the probability that it is in a bad patch becomes sufficiently high, the animal may choose to move on. Many optimal foraging models aim to derive a precise prediction about when an animal should move from one patch to another.
In models of this sort, talk of an animal’s ‘knowledge’ is intended behavioristically. Saying that the animal ‘knows’ that 1/3 of food patches are good means that the animal behaves as if it knows this, as manifested in its behavior. If the environment can be in a number of different states, and the appropriate action depends on the state, then to behave adaptively an animal will need to be sensitive to the states’ probabilities of occurrence. If the animal chooses between actions according to maximization of expected reproductive output—one standard definition of adaptive behavior—then it is behaving as if it knows the true probability distribution on the states, knows the payoffs, and is able to calculate expectations.
Understanding ‘knowledge’ in this behavioristic way may seem quite different from how the notion is understood in epistemology. But in fact many Bayesian decision theorists, notably Savage (Reference Savage1954), have argued for a behavioristic interpretation of probabilistic beliefs, according to which an agent’s subjective probabilities (and utilities) are derived from their choices between uncertain prospects, which are in principle observable; this interpretation is orthodox in contemporary economics, if not in contemporary epistemology. So in fact, evolutionary biologists’ use of Bayesian concepts to model animal behavior is not such a radical departure.
3. A Simple Example
To fix ideas, consider a simple example. An organism is foraging for food in a predator-strewn area. Predators are of two types, snakes and leopards. If no predator is present, the best thing to do is stay put and forage. If a snake is present, the best thing to do is to climb a tree. If a leopard is present, the best thing to do is flee. Both climbing and fleeing are costly in terms of time and energy, to different extents. Payoffs for the three actions, as a function of the state of the world, are shown in table 1; these payoffs are measured in increments of biological fitness (i.e., number of offspring).
| No Predator | Snake | Leopard | |
|---|---|---|---|
| Stay put | 10 | 0 | 0 |
| Climb | 5 | 4 | 0 |
| Flee | 6 | 1 | 2 |
What action should the animal choose to maximize its expected payoff? That depends on the probabilities of the three states of the world. Suppose that the probabilities of the three states are
(These probabilities can be thought of as the relative frequencies with which each state occurs in the relevant local ecology.) Then, the expected payoff from each action is
So the evolutionarily optimal action is to stay put. Organisms choosing to stay put will on average leave more offspring than those choosing either of the other two actions. If the organisms’ choice behavior has been optimized by natural selection, or if it has learned which action is optimal, then it will stay put.
Now suppose that before choosing an action, the organism receives a signal that indicates whether a predator is present. So the signal has two values: ‘safe’ and ‘unsafe’. The signal is perfectly reliable (i.e., indicates ‘unsafe’ if and only if a predator is present) but cannot discriminate between leopards and snakes. The organism’s choice of action may depend on which signal is received. So the organism needs to have a ‘policy’, that is, a specification of which action to take for each value of the signal. There are 9 (
) possible policies, in this example. Let us consider three policies in particular.
Suppose first that the organism ignores the signal and always chooses the action optimal for the situation in which no signal is received (i.e., stay put). So the organism’s policy is ‘if safe, stay put; if unsafe, stay put’. Let us denote this policy ‘Ignore’. The policy may seem unpromising, intuitively, but it could reflect the organism’s cognitive limitations. An organism incapable of attending to the signal, or lacking behavioral plasticity, might use a policy like Ignore.
Second, suppose that the organism does attend to the signal and adopts a maximin strategy; that is, it chooses the action that maximizes the minimum payoff it will receive, in the light of the signal. So if the safe signal is received, it chooses to stay put, obviously. If the unsafe signal is received, it chooses to flee, as this guarantees it a payoff of at least 1. So its policy is ‘if safe, stay put; if unsafe, flee’. Let us call this policy ‘Maximin’. The policy reflects a high degree of risk aversion. By choosing to flee rather than climb when the unsafe signal is received, the organism forgoes a possible payoff of 4 in order to definitely avoid a payoff of 0.
Third, suppose that the organism behaves like a Bayesian. On receipt of a signal, the organism chooses an action that maximizes its conditional expected payoff, given the signal.Footnote 1 So if the safe signal is received, the organism chooses to stay put—as the conditional expected payoffs are then 10, 5, and 6 for stay put, climb, and flee respectively. (This is because the conditional probability of the no-predator state, given the safe signal, is 1.) What if the unsafe signal is received? The conditional probabilities of the no-predator, snake, and predator states, given an unsafe signal, are 0, 2/3, and 1/3 respectively. The conditional expected payoffs for staying put, climbing, and fleeing are then 0, 8/3, and 4/3 respectively, so the organism will choose to climb. Its policy is therefore ‘if safe, stay put; if unsafe, climb’. Let us call this policy ‘Bayes’.
Which of our three policies—Ignore, Maximin, and Bayes—is the best, from an evolutionary point of view? To answer this question, we need to compute the expected payoffs accruing to an organism that uses each of the policies. Consider first Ignore. Since an organism using Ignore chooses to stay put whether or not a signal is received, its expected payoff is
What about Maximin? An organism using Maximin stays put if a safe signal is received but flees otherwise. With probability 1/2, no predator is present, so the safe signal is sent, so the organism stays put and earns a payoff of 10. With probability 1/3, a snake is present, so the unsafe signal is sent, so the organism flees and earns a payoff of 1. With probability 1/6, a leopard is present, so the unsafe signal is sent, so the organism flees and earns a payoff of 2. Its expected payoff is, therefore,
By a similar logic, the payoff to an organism using the Bayes policy is
Therefore, the Bayes policy yields the highest expected payoff. If all three policies are found in a population, natural selection will favor the Bayes policy over the other two. This suggests, obviously in a preliminary way, that evolution might produce organisms that behave as if they were rational Bayesian agents, whose subjective probabilities over the states of nature match the objective frequencies, and who incorporate new information by conditionalization. A stronger argument for this conclusion would need to show that no conceivable policy does better than the Bayes policy; see section 4.
4. Cannot Do Better than Bayes
The framework we adopt directly generalizes the previous example. There is a finite set S of states of nature; 
. There is a finite set U of actions that the organism may perform. The payoff from an action depends on the state of nature. The payoff from action 
if the state of nature turns out to be 
is 
. As before, payoffs are measured in increments of biological fitness.
There is a finite set E of possible signals, or items of evidence, that the organism may receive; 
. In the example above we assumed that the signal set partitioned the states of nature, that is, each state of nature was compatible with exactly one signal, but no such assumption is made here. There is a joint probability distribution p on 
, reflecting the frequency with which each (state, signal) pair occurs. The marginal probability of state 
is 
, and that of signal 
is 
. We assume that 
for all i and 
for all j; that is, each state occurs with positive probability, and similarly for each signal.
An organism’s ‘policy’ specifies an action 
for every possible signal 
. Thus, a policy is simply a function from E to U. The set of all policies is denoted X. For any policy 
, we let 
be the action specified by that policy when signal 
is received; for convenience, we write 
as 
.
Consider an arbitrary policy 
. What is an organism’s expected payoff from using policy x? Suppose first that the true state of nature is 
. Then, the expected payoff from policy x equals
The justification for (1) is clear: 
is the probability that signal 
is received given that the state of nature is 
; 
is the payoff from choosing action 
—the action specified by policy x when the signal received is 
—in state of nature 
. Expression (1) is thus the expected payoff to an organism using policy x when the true state is 
. Taking the expectation of (1) across states of nature thus gives us the overall expected payoff to policy x:
Now recall the Bayes policy discussed above. In our simple example above, there was a unique Bayes policy since for each signal there was a unique action that maximized conditional expected payoff given the signal. But in the general case this need not be so: for some signals, there may be more than one action satisfying this maximization constraint. So we need to consider the class of Bayes policies 
, where each 
is a policy that, for each signal 
, chooses an action 
that maximizes conditional expected payoff given 
.
We wish to show that a policy is evolutionarily optimal if and only if it is a Bayes policy; as before, ‘evolutionarily optimal’ means ‘earns an expected payoff greater than or equal to every other policy’. To show this, consider any Bayes policy 
. Substituting b for x in expression (2) thus gives us the expected payoff to a Bayes policy:
Our first task is to show that expression (3) is greater than or equal to (2); that is, policy b has expected payoff greater than or equal to policy x.
Since b is a Bayes policy, we know that it satisfies the following condition: for every signal 
,
for all actions 
. Applying inequality (4) to the particular action 
gives
Applying Bayes’s theorem and dividing across by 
,
Summing over all the signals,
Reversing the order of summation,
But the left-hand side of (8) is the expected payoff to policy b (eq. [3]), while the right-hand side is the expected payoff to policy x (eq. [2]). So policy b is evolutionary optimal. Since nothing has been assumed about policy b except that it is a Bayes policy, we can conclude that every Bayes policy is evolutionarily optimal.
To show the converse, consider another policy y, which is not a Bayes policy. Thus, for some signal 
,
Applying Bayes’s theorem,
Since b is a Bayes policy, we know from (6) that for all signals 
:
From (10) and (11), we can deduce that, summing over all the signals,
Reversing the order of summation,
But the left-hand side of (13) is the expected payoff to policy y, while the right-hand side is the expected payoff to the Bayes policy b. So any policy that is not a Bayes policy earns a strictly lower expected payoff than a Bayes policy, so it cannot be evolutionarily optimal. Therefore, an evolutionarily optimal policy must be a Bayes policy.
Taken together, (8) and (13) tell us that a policy is evolutionarily optimal if and only if it is a Bayes policy. So organisms that implement Bayes policies will leave more offspring, on average, than ones that do not. Over time, we would expect the non-Bayes policies to be eliminated by natural selection.
5. Discussion
What exactly does the foregoing argument show? The argument seems to supply an evolutionary basis for Bayesian updating, by showing that it follows from the standard Darwinian assumption that animal behavior is well adapted and, hence, designed to maximize the animal’s expected reproductive output. However, some care is needed here. Typical discussions of Bayesian updating ask how an agent does or should update its epistemic state on receipt of information, where an epistemic state is represented by a prior probability distribution over some appropriate algebra. But our argument above makes no mention of ‘epistemic states’ or ways of updating them, and it makes no assumption that the organisms in question are capable of being in such states. We have assumed only that the organisms are behaviorally plastic and capable of modifying their behavior in response to an environmental signal.
To get from this behaviorist starting point to something more epistemic, we need to consider how an organism might implement an evolutionarily optimal policy, as characterized above. One possibility is this: the organism perates with an internal ‘belief-like’ representation of the environment and an internal ‘desire-like’ representation of the outcomes (or payoffs), and it uses a ‘choice rule’ to determine which action to pick, depending on which belief and desire states it is in. On receipt of a signal, the organism uses an ‘update rule’ to go from one belief state to another. Given a generic psychological makeup of this sort, which is presumably applicable to at least some higher animals as well as humans, an evolutionarily optimal policy is straightforward to implement.
To see this, suppose we take a ‘belief state’ to be a subjective probability distribution over the states of nature and signals and a ‘desire state’ to be a real-valued utility function on the outcomes (or action-state pairs). In particular, suppose that the organism’s initial belief state, before receiving a signal, is equal to the true prior distribution p, and its utility function is equal to the true payoff function V (or an affine transformation thereof). Suppose that the organism’s choice rule is ‘maximize expected utility, relative to my current belief state’ and that its update rule is Bayesian conditionalization. Then, it follows that the organism will implement an evolutionarily optimal policy as characterized in section 4; this can be seen by direct inspection of expression (3).
What could license the assumptions that the organism’s initial belief state is correct (i.e., matches the true distribution on the states) and that its utility function matches the true payoff function? A short answer is ‘evolution by natural selection’. Organisms often have information about the environment ‘preprogrammed’ into their genome, as a result of natural selection, and are often equipped with ‘appropriate’ desires for surviving and reproducing (e.g., many animals are born with an innate ability to determine who their close relatives are and innate preferences for some food items over others). From an adaptationist perspective, therefore, the assumptions in question do not seem unreasonable.
Given these assumptions, our argument shows that an organism will implement an evolutionarily optimal policy if it chooses between actions in accordance with expected utility maximization relative to its current belief state and always updates its belief state by the rule of conditionalization. Therefore, two standard principles of Bayesian rationality fall out of the requirement of evolutionary optimality. This suggests that, other things being equal, natural selection should lead to organisms that satisfy these Bayesian principles.
This conclusion needs qualifying in a number of ways. First, the inference from ‘is evolutionarily optimal’ to ‘is likely to have evolved’ can obviously be questioned; this is a well-known issue in the literature on adaptationism. Second, defining optimality in terms of maximal expected reproductive output raises some tricky theoretical issues discussed in the next section. Third, our argument does not show that the only way for an organism to implement an evolutionarily optimal policy is to operate with internal belief-like and desire-like representations that combine and update in accordance with Bayesian principles. This last point merits some brief discussion.
It is well known from the work of Gerd Gigerenzer that adaptive behavior may be produced by simple heuristics, which are computationally less demanding than optimality calculations (e.g., Gigerenzer, Todd, and Pachur Reference Gigerenzer, Todd and Pachur2011). This general point applies to our problem of choosing an optimal policy. Even if there are many states of nature and signals, an organism might still be able to implement an optimal policy by a simple heuristic such as ‘if signal x is received, run away; otherwise, stay’, depending on the payoff function. And in fact, many simple animals exhibit behavioral plasticity in response to environmental cues but presumably do not operate with a belief-like representation of the environment at all. Of course in a purely ‘as if’ sense, these animals are behaving like Bayesian agents so long as they are implementing an optimal policy, as our argument shows. But we should not blur the distinction between organisms that implement an optimal policy by using a heuristic and ones that operate with internal belief-like and desire-like representations and are thus capable of satisfying Bayesian principles in more than an ‘as if’ sense. An important question is when evolution will lead to organisms of each type. This question is not addressed by the foregoing analysis; see Sterelny (Reference Sterelny2003) for a sketch of a possible answer.
6. Relation to the Brown/Maher Argument
It is worth relating our evolutionary argument to a related argument found in the rational choice literature. Both Brown (Reference Brown1976) and Maher (Reference Maher1992) try to justify Bayesian conditionalization by arguing that it follows from the injunction to maximize expected utility. They consider an agent with a prior subjective probability distribution over a set of states of nature, a set of possible actions, and a utility function defined on the (state, action) pairs. The agent receives a signal, updates to a new probability distribution, then chooses an action that maximizes expected utility relative to this new distribution. Brown and Maher then show that in order to maximize expected payoff with respect to the agent’s prior distribution, their update rule should be conditionalization. They interpret this to mean that rationality requires an agent to update in a Bayesian way.
Formally, the Brown/Maher result is a special case of our own. For the Brown/Maher argument identifies a particular way of implementing an optimal policy, for an agent who uses an update rule to go from belief state to belief state and a choice rule to go from belief state and utility function to choice of action. Brown and Maher assume that the choice rule is expected utility maximization relative to the updated state and, then, show that the update rule should be conditionalization. Our result is more general in that it does not assume from the outset that the organism implements its policy by using an (update rule, choice rule) combination at all. However, it follows directly from our result that if an organism implements its policy this way, and if its choice rule is ‘maximize expected payoff relative to the updated state’, then its update rule should be conditionalization—which is the Brown/Maher result. This follows directly from the characterization of an optimal policy in expression (3).
To see that our result is stronger than the Brown/Maher result, notice that theirs leaves open the possibility that an agent could earn a higher expected payoff, relative to the prior distribution p, by using an update rule other than conditionalization and a choice-of-action rule other than maximize expected payoff. This possibility is not ruled out by the Brown/Maher argument since they take for granted that the choice-of-action rule is maximize expected payoff. However, our result rules the possibility out. Since the combination (conditionalization, expected payoff maximization) implements an optimal policy, it follows that no other combination can do better, even if choice-of-action rules other than expected payoff maximization are permitted.
7. A Qualification
The foregoing argument assumes that expected reproductive output is the right criterion of evolutionary success. But in fact this is not always the case. It is well known that in certain situations, the expected performance of a strategy (or phenotype) is not the sole determinant of whether it will evolve; variability in performance can also matter (Gillespie Reference Gillespie1977; Frank and Slatkin Reference Frank and Slatkin1990; Seger and Brockmann Reference Seger and Brockmann2002). How does this complication affect the foregoing argument?
Expected reproductive output is an appropriate criterion of evolutionary success (or ‘definition of fitness’) when two conditions are met. First, the population must be large, and second, the probability distribution over states of nature that any organism faces must be independent across the members of the population. These conditions jointly ensure that the average reproductive success of the cohort of organisms using a given strategy will be very close to that strategy’s expected success. The large population condition will often be satisfied, but the independence condition is more problematic.
In the predator example of section 3, the independence condition seems reasonable. The probability that organism A encounters a snake while foraging may well be independent of the probability that organism B does; risks of this sort are sometimes called ‘idiosyncratic’. But in other cases risks will be aggregate. The weather is an obvious example, as it affects many organisms at once. If there is a 5% chance of a harsh winter, this risk will obviously not be independent across all members of a biological population. Real organisms probably face both idiosyncratic and aggregate risks, in varying combinations depending on the context.
Where risks are aggregate, the strategy with the highest expected reproductive output need not be evolutionarily optimal (McNamara Reference McNamara1995; Robson Reference Robson1996). The variance in output matters too, and selection will penalize strategies with a high variance. In such a case, our simple argument for the evolutionary optimality of Bayesian updating does not apply. This suggests that in theory, organisms whose behavior is mediated by internal belief-like and desire-like states might do best to use Bayesian updating some of the time but not always. In a circumstance in which risks are idiosyncratic, as in our predator example, an organism should incorporate new information by Bayesian updating, assuming that its choice-of-action rule is ‘maximize expected payoff in the updated state’. But where risks are aggregate, as in the weather example, a non-Bayesian update rule might be evolutionarily superior. However, to implement this selectively Bayesian strategy would require being able to distinguish aggregate from idiosyncratic risks. It seems unlikely that organisms can do this, but it is an empirical question.
