1. Introduction
Cognition is generally costly to maintain, can lead to errors, and is not always necessary. This presents an evolutionary puzzle: Why did cognition evolve? One intuitive answer is that cognition enables adaptation to a variable or complex environment (Godfrey-Smith Reference Godfrey-Smith1996). Other organisms are an important source of environmental complexity, and many of our cognitive processes are directed toward one another, which suggests that social interaction may be particularly important for the evolution of cognition. This is the general notion behind the social intelligence hypothesis (Humphrey Reference Humphrey, Bateson and Hinde1976): cognition evolved to help us navigate our social world. We might go further, however, and investigate what types of social interaction are important for the evolution of cognition. Communication is an obvious candidate. Cognition seems to enable better use of signals and information—both sending and receiving. Here, I address this possibility by exploring some evolutionary models of the simplest instances of language and cognition: signaling games and behavioral plasticity.
Plasticity is nicely exemplified by rudimentary behavioral learning. In recent years, evolutionary game theory has been used to examine the evolution of learning in social settings (see, e.g., Harley Reference Harley1981; Maynard Smith Reference Maynard Smith1982; Dubois, Morand-Ferron, and Giraldeau Reference Dubois, Morand-Ferron and Giraldeau2010; Katsnelson et al. Reference Katsnelson, Motro, Feldman and Lotem2012; Smead Reference Smead2012). In the context of signaling, game theoretic models have shown that plasticity does not generally persist in common-interest signaling games: plasticity can help a population reach efficient signaling systems, but it is ultimately eliminated by natural selection in favor of nonplastic behavior (Zollman and Smead Reference Zollman and Smead2010). While common-interest signaling may occur frequently in nature, many signaling games are not common interest. When interests of signaling parties diverge, deception becomes possible and may be a probable evolutionary outcome (Skyrms Reference Skyrms2010). This article presents a set of models for the evolution of a simple reinforcement learning rule in signaling games with varying degrees of common interest. As we will see, the best case for the evolution of plasticity involves partial common interest, where plastic types can gain an advantage by learning when to deceive and when to trust others.
In what follows we will make use of two-population evolutionary models, which are most naturally interpreted as representing interactions between species. However, I will argue that these models need not be interpreted in this way and may also represent interactions among conspecifics or even sets of interacting cells within an organism. Furthermore, although common-interest signaling is not generally conducive to the evolution of plasticity, the two-population model reveals possibilities not seen in one-population settings.
2. The Model
The Lewis sender-receiver game has become a standard model for studying the evolution of signaling (Lewis Reference Lewis1969; Skyrms Reference Skyrms1996). In this game, there are two players: the sender and the receiver. The sender observes some state (s) of the world and sends a message (m) to the receiver. The receiver observes the message, but not the state, and must choose an action (a). The payoffs to the sender and the receiver depend on the state-act combination.
For tractability, I focus only on two-state, two-message, two-act signaling games with equiprobable states. A sender’s strategy tells her which message to send in which state. A receiver’s strategy tells her which act to do in response to each possible message. The possible strategies are listed below.
S1. Send m 1 in s 1 and m 2 in s 2.
S2. Send m 2 in s 1 and m 1 in s 2.
S3. Always send m 1.
S4. Always send m 2.
R1. Do a 1 when m 1 and a 2 in m 2.
R2. Do a 2 when m 1 and a 1 in m 2.
R3. Always do a 1.
R4. Always do a 2.
Strategies S1, S2, R1, and R2 are referred to as “separating” strategies; the others are “pooling” strategies. Let 
be the sender’s payoff when strategy i plays strategy j, and 
is the receiver’s payoff.
2.1. Evolution in Two Populations
I use the two-population replicator dynamic to model evolution (Taylor and Jonker Reference Taylor and Jonker1978; Hofbauer and Sigmund Reference Hofbauer and Sigmund1998). Individuals in a sender population are randomly paired with individuals in a receiver population to play a signaling game. Populations are continuous, and states are represented as vectors 
, where xi tracks the frequency of type i in the population and there are N types. Types in the population correspond to strategies in the signaling game. Let X represent the sender population and Y the receiver population. The fitness for each type can be calculated as 
and 
for senders and receivers, respectively. Evolution in each population occurs according to a set of differential equations: 
for the sender population and similarly for the receiver population.
It will be convenient at several points to employ computer simulations of a discrete-time version of the two-population replicator dynamic. The discrete-time dynamic uses the difference equation 
and shares many properties with the continuous time version. Where simulation results are reported, results are aggregated from 1,000 independent simulations with an initial population chosen at random from the population space with a uniform distribution. All simulations were run for 106 generations unless noted otherwise.
Several of the equilibria identified below are only weakly stable (Lyapunov stable) in the replicator dynamic and, as a result, may be partially influenced by drift and the stochastic nature of selection. For this reason, it will be helpful to also include mutation in the model. A standard approach is to assume that mutation occurs uniformly across strategies with probability 
. In the discrete-time replicator dynamic this amounts to 
.
2.2. Including Plasticity
An additional plastic type L will also be included in the sender and the receiver populations. It is assumed that interactions between pairs of individuals are repeated and that L is able to recognize individuals, giving this type an opportunity to adapt within each interaction.Footnote 1 More specifically, plastic individuals will be Roth-Erev reinforcement learners (Herrnstein Reference Herrnstein1970; Roth and Erev Reference Ristau and Ristau1995; Beggs Reference Beggs2005). These learners begin each repeated interaction by selecting a strategy stochastically according to an initial set of weights (1.0 for each strategy), playing that strategy, then observing the payoff and updating the weights by an amount equal to the payoff received. This process is repeated for a fixed number of rounds (10,000 in this case). At each iteration, the probability of choosing a strategy is equal to the cumulative payoffs received when choosing that strategy (plus the initial weight) divided by the total payoffs received to that point (plus all initial weights).
Evolutionary analysis proceeds in two phases. The first phase involves constructing a type game that includes L in addition to all pure-strategy types. For contests between non-L types, payoffs of the type game are identical to payoffs of the original game. In contests with L types, payoffs are estimated using 105 independent numerical simulations of learning in a repeated interaction and calculating the average payoff per round of interaction. The second phase of analysis is to apply the replicator dynamic to the type game and assess whether and when plasticity will evolve and persist.
Plasticity is frequently assumed to carry some cost. Roth-Erev reinforcement learning already carries the cost of inefficiency from possible errors and time to learn optimal behavior. Since we are only using the first 10,000 learning trials to estimate payoffs, this inefficiency will be captured in the type game. However, there may also be metabolic or other fitness costs associated with the ability to learn. For this reason, we will consider the evolutionary prospects for plasticity both with and without an exogenous cost (c).
3. Plasticity with Common-Interest Signaling
To represent perfect common interest, suppose both players receive a payoff of 1 when a 1 is chosen in s 1 and when a 2 is chosen in s 2, but receive 0 otherwise. Evolution in common-interest signaling games can be complex and does not always lead to perfect communication (Huttegger and Zollman Reference Huttegger, Zollman, Benz, Ebert, Jäger and van Rooij2011). In the simple case we are considering, however, evolution always leads to a signaling system: either (S1, R1) or (S2, R2).
Zollman and Smead (Reference Zollman and Smead2010) have investigated the evolution of plasticity in settings of common-interest signaling using one-population models. They find that learners can initially increase in frequency within a population but invariably get eliminated by selection. Plasticity is useful when there is variation in the signaling strategies used by members of the population. Once plasticity becomes common, however, it is no longer worth the cost of error—a strategy that signals in a fixed way will be accommodated by plastic individuals and perfectly coordinate with themselves.
It is known that the results of two-population models and one-population models do not always coincide, and this turns out to be the case here. Including learners in, say, the sender population does not change the stability of either signaling system. If the receiver population always plays a separating strategy, there is no benefit to behavioral flexibility, and the corresponding (fixed) signaling strategy does best. However, if the receiver population includes enough variation, it pays to be plastic. Type L can adopt S1 when interacting with R1’s and S2 when interacting with R2’s.Footnote 2 If there is a large degree of initial variation in the receiver population, this can lead to the evolutionary fixation of learning in the sender population even when there is a small exogenous cost to learning (see fig. 1). This also holds true if L is included in both populations. Without mutation, fixation of plasticity in one of the two populations occurred in 26.6% of simulations when c = 0 and 19.4% when c = .05. Thus, the results of Zollman and Smead (Reference Zollman and Smead2010) do not generalize to the two-population case. Variation in the signaling behavior in one population can lead to the evolution of plasticity in the other population.
Replicator dynamic in a two-population common-interest sender-receiver model including learners in the sender population: c = .05. Pooling strategies have been removed for visual representation. Interior points represent population distributions for the sender population (in the L, S1, S2 space) and the receiver population (in the R1, R2 space). Arrows represent the direction of evolutionary change. Gray points are unstable rest points; black points are (Lyapunov) stable. Phase portraits were created using Mathematica 9.
There are some important caveats to this result, however. First, the population states that include plastic individuals are not strongly stable. Once learners dominate one population, there is no fitness difference between the separating strategies in the other population because L will accommodate each equally well. If evolutionary drift leads to a predominance of either separating type, the corresponding type in the other population will outperform the plastic L. Second, it is not possible to maintain plasticity in both the sender and receiver populations simultaneously. The reason is that only one of the two players in each interaction needs to be plastic in order to ensure a high degree of signaling success. If one population is dominated by plastic individuals, fixed separating strategies will win out in the other population.
Additionally, simulations that include mutation show that plasticity does not persist. With 
and 
, all simulated populations reached approximate signaling systems with a minimal frequency of L. Since mutation rates are positive, evolution never fixates on the signaling systems but gets very close (e.g., ≈.993 for 
).
However, if we examine simulated populations in the relatively short run with low mutation, we can see that plasticity occasionally persists for some time before being driven out. With no cost and 
, plasticity made up more than .95 of one population in 22% of simulations at 103 generations. This figure dropped to 11.2% at 104 generations. Including costs or increasing the mutation rate greatly reduced the prospects for plasticity. A large frequency of plastic types in one population, when observed, was almost always accompanied by a significant mixture of nonplastic separating types in the other population. Although plasticity may not be prominent in the limit of the evolutionary processes, it can have a significant presence in the relatively short term.
4. Plasticity with Opposed-Interest Signaling
Now suppose we have the same setup as before, but where the receiver’s payoffs are inverted. The players’ interests are now directly opposed: the sender wants the receiver to do a 1 in s 1 and a 2 in s 2, but the receiver wants to do the opposite. Opposed-interest interactions are common in predator-prey interactions, and the evolutionary dynamics in such interactions frequently show cycles similar to the well-known Lotka-Voltera models. In some cases of zero common-interest signaling games, the cycles are not regular but chaotic (Wagner Reference Wagner2012).
With directly opposed interests and including only S1, S2, R1, and R2, the population will cycle around the point where all strategies are represented equally. These cycles go through phases in which the senders systematically mislead the receivers and phases in which the receivers take advantage of the information contained in the senders’ signals. If we include the other sender and receiver strategies, similar persistent cycles are observed, but they include some proportion of pooling strategies as well. Skyrms (Reference Skyrms2010) refers to the states in which the sender is manipulating the receiver as cases of “deception.”
Skyrms defines a deception in informational terms as a signal that raises the probability of a state that is not the true state to the benefit of the sender and the detriment of the receiver. This definition is not uncontested (Godfrey-Smith Reference Godfrey-Smith2012). For our purposes here, however, I take Skyrms’s definition for granted and use the term “deception” in his sense. Deception is only possible in signaling games without perfect common interest; games with diverging interests are known to provide settings more conducive to the evolution of plasticity (Smead Reference Smead2014).
Restricting the strategy space to separating strategies and including L in the sender population results in the evolution of plasticity (see fig. 2). This result is not as fragile as the evolution of plasticity in the common-interest case. Although the same Lyapunov-stable equilibrium structure is observed, these points are now the only attracting equilibria in the system. If drift leads to the predominance of one receiving strategy, evolution will favor one of the two nonplastic sender strategies. However, once such a nonplastic strategy invades, selection will cause the receiver population to shift and thus restore L’s advantage. This restricted signaling game provides an interaction environment that strongly favors learning.
Replicator dynamic in a opposed-interest sender-receiver game including learners in the sender population: c = .05. Pooling strategies are omitted.
Including pooling strategies, however, shows that this result is not robust. Simulations both with and without mutation show that if plasticity carries a cost (c = .05), L never persists in large frequencies when all strategies are included. In every case, the populations are composed of pooling strategies almost entirely. The reason for this is that pooling strategies cannot be exploited by L, cost nothing to learn, and outperform separating strategies in the presence of learners. Costly plasticity does poorly in the short run as well. After 1,000 generations with no mutation or low mutation rates (
), no simulations had significant numbers of plastic strategies in either population. The decline of costly plasticity happens more rapidly in the case of opposing interests than in the case of common interest, and separating strategies are driven out as well. In opposed-interest signaling games, costly plasticity and deception work against one another and neither evolves.
Costless plasticity fares much better. For 
, both populations converge to a state with ≈.599 L, .2 of each pooling type, and a small proportion of separating strategies maintained by mutation: 
produces similar results. The constant small presence of separating strategies is what creates the advantage for L over pooling strategies. The possible, although rare, advantage of deception—and the advantage of avoiding being deceived—allows learners to persist at high frequencies. Despite this dynamic, these populations do not have a particularly high degree of deceptive behavior. When two learners interact, they often learn to adopt pooling strategies, or the receiver gets the upper hand and avoids defection. Consequently, the overall level of deceptive behavior in these populations is comparable to that of populations with no learners. So, although deception is where plasticity gains its advantage, the presence of L does not lead to a substantial change in the frequency of deception.
5. Plasticity with Partial Common Interest Signaling
Neither common nor opposed-interest signaling is particularly conducive to the evolution of costly plasticity. However, many instances of signaling games found in nature probably lie somewhere between these two extremes. Indeed, models of costly signaling—a central focus in the study of animal communication—often involve partial common interest (Zahavi Reference Zahavi1975; Maynard Smith and Harper Reference Maynard Smith and Harper2003; Searcy and Nowicki Reference Searcy and Nowicki2005). In games with partial common interest, there are some states in which successful communication is in the interests of both parties and other cases in which there is an incentive to deceive.Footnote 3
Consider a signaling game with the payoffs in table 1. In this case, when the state of the world is s 1 interests are aligned, and both players want the receiver to do a 2. When the state is s 2, the sender would prefer the receiver to do a 2, while the receiver would prefer to do a 1. If the signals are costless and the states are equiprobable, evolution will always lead to a pooling equilibrium where the sender population mixes between separating and pooling strategies but pools often enough that the receivers always ignore the signal and choose a 1. However, when signal costs of the right kind are introduced, the evolutionary dynamics change significantly.
| a 1 | a 2 | |
|---|---|---|
| s 1 | 0, 0 | 5, 1 |
| s 2 | 1, 5 | 2, 2 |
Suppose that it is cheap to send m 1 in s 1 and m 2 in s 2 and that it costs .1 to send m 2 in s 1 or m 1 in s 2. This may occur because there is a natural salience to the signals that it takes some effort to disguise. Or, this may be because certain separating behavior generates some kind of payoff outside of the game taking place. Whatever the cause, a consequence of this cost structure is that, when facing a receiver who is ignoring the signal, it is better to adopt the separating S1 strategy. With these costs, there is a unique Nash equilibrium of the game: the sender plays S1 with probability 2/3 and S3 with probability 1/3, the receiver plays R2 with probability 1/10 and R3 with probability 9/10. This point is also the only interior rest point in the replicator dynamic—populations will fall into orbits around this equilibrium (see fig. 3). With 
, simulations converge to a point near the equilibrium with frequencies (.628138, .006637, .358536, .006690) of (S1, S2, S3, S4) and (.006579, .101911, .885286, .006224) of (R1, R2, R3, R4), respectively. With 
, simulations fall into to a regular orbit about the equilibrium with average frequencies near the equilibrium point.
Replicator dynamic in a partial common interest sender-receiver game. Strategies S4, R1, and R4 omitted for visual clarity.
Deception is a regular occurrence in these populations. When S3 interacts with R2, S3 raises the probability that the state is s 1 when it is not, and R2 reacts by choosing an action that benefits the sender at the cost of the receiver. To determine whether this more complicated interaction is conducive to the evolution of plasticity, we can introduce L into the model as before. This generates the type game represented in table 2. There are three Nash equilibria in this type game:
1. X ≈ (.1105, .8895, 0, 0, 0) and Y ≈ (.0533, 0, 0, 0, .9467),
Table 2.Type-Game Payoffs for a Partial Common Interest Sender-Receiver Game with Learners
R1 R2 R3 R4 L S1 1, 1 3, 3 .5, 2.5 3.5, 1.5 2.167, 2.779 S2 2.9, 3 .9, 1 .4, 2.5 3.4, 1.5 2.06, 2.779 S3 .45, 2.5 3.45, 1.5 .45, 2.5 3.45, 1.5 .686, 2.421 S4 3.45, 1.5 .45, 2.5 .45, 2.5 3.45, 1.5 .686, 2.421 L 3.238, 2.048 3.265, 2.088 .464, 2.500 3.452, 1.500 1.679, 2.433 Note Payoff with L is approximate (to nearest .001) and was estimated using 105 independent simulations.
2. X ≈ (0, .6353, 0, 0, .3647) and Y ≈ (.5299, 0, 0, 0, .4701),
3. X ≈ (.6095, 0, 0, 0, .3905) and Y ≈ (0, .6481, 0, 0, .3519).
Simulations reveal that only the second and third are evolutionary significant. For 
, populations converge to one of two points that are near the second and third equilibria. For 
, populations fall into an orbit around one of those equilibria. In all cases, although learners are not predominant, they are sustained in both populations at relatively high frequencies. This holds with both high and low mutation rates as well as when learning carries an additional exogenous cost.Footnote 4
The presence of learners also greatly increases the frequency of deceptive signaling. Without learners, the rate of successful deception at the equilibrium is relatively low—.033 of signals—due to the high proportion of the pooling strategy R3. Note that when L is included, the equilibria do not include any pooling types. Plastic types learn when to use deceptive signals as a sender and when to trust signals as a receiver. Even with mutation, the advantage of L keeps pooling types at small frequencies (less than .06 of any pooling type) and creates populations that are much more responsive to signals. This responsiveness is accompanied by higher degrees of deception, often by learners against separating receivers. Type L also can learn that separating senders should be trusted. In this game, plasticity gains an advantage by learning when to bluff and when to take a signal seriously.
6. Discussion and Conclusion
We have considered the evolution of plasticity in three cases of signaling interactions. The best case for the evolution of plasticity involves partial common interest where both honest communication and deception are incentivized. Evolution in this setting creates a varied and complex population where plasticity thrives. This result coheres with the environmental complexity thesis on the evolution of cognition. It is also in line with the idea that social settings involving both cooperation and conflict are important for the evolution of cognition more generally (Byrne and Bates Reference Byrne and Bates2007; Sterelny Reference Sterelny2012).
Where should we expect to find signaling that is conducive to the evolution of plasticity? Paradigm cases of common-interest signaling might be communication among close relatives or communication among an organism’s cells. Interests may diverge if signalers are not closely related or are members of different species. Thus, it may be reasonable to suspect that the important interactions for the evolution of plasticity are those among unrelated conspecifics or those that are interspecific. This also seems to be the case with common-interest signaling: two-population settings are more conducive to the evolution of plasticity than one-population settings. It is natural to interpret the two populations as representing two species, which suggests that interspecific interactions may be particularly important for the evolution of plasticity.
However, it is not necessary to interpret two-population models as representing interspecific interaction, nor is it true that interactions with kin and within-organism interactions are always common interest. Two-population models represent situations in which interactions are directed between (but not within) groups and in which the relevant standard of success is within (but not between) groups. This can occur within a single species (e.g., conditional interaction between different demes, interaction between sexes, or interaction between generational groups). It may also occur within an individual organism (e.g., an organism’s cells are interacting with pathogens or symbionts). These cases are also ones that may involve opposed or partially aligned interests. Of course, real interactions usually involve both within- and between-population aspects, often simultaneously. No one would claim that two-population models capture most of the complicated nature of real interactions. Nonetheless, they can capture an important aspect of evolutionary interaction, which can occur within or between organisms, conspecific or heterospecific.
Along these lines, deception seems to occur in all areas of interaction. HIV is an example of a host-pathogen interaction that has been called “deceptive” (Krishnamoorthy et al. Reference Krishnamoorthy, Bess, Preston, Nagashima and Mahal2009). Stomatopods’ aggressive display while molting, which renders them unable to fight, is an example of conspecific deception (Steger and Caldwell Reference Steger and Caldwell1983). “Distracting displays by piping plovers are a case of inter-specific deception” (Ristau Reference Steger and Caldwell1991). Interestingly, stomatopods and piping plovers are sometimes noted for their apparent cognitive abilities. HIV also displays structural plasticity (Korkut and Hendrickson Reference Korkut and Hendrickson2012).
The models explored here show that signaling games of partial common interest can generate complex and varied interaction settings. In such a case, plasticity is able to invade populations and persist in relatively high frequencies. The reason for this is that plastic individuals are adept at both deceiving and avoiding being deceived. These models can be applied to a wide range of possible biological interactions ranging from within-organism signaling to interspecific signaling. It is important to note that plasticity may be less likely to occur at the very small scale due to the complexity required for plastic responses. This aside, the key to the evolution of plasticity in signaling games seems to be the character of the signaling interaction rather than where the interaction takes place.
and 
, respectively. With c = .05, the average proportion of L was .374 and .340.
