1. Introduction

Hume’s skepticism was based, in part, on what he believed to be an inability of human observers to ever go beyond their perceptual experiences. How can an observer determine the accuracy of perception when her only access to the external world is mediated by perception itself? A related question may be asked about not the qualities of perception but rather the perceptual categories of biological actors. How can people reach beyond perception to determine whether and to what degree perceptual categories—sweet, green, cool—track properties of the world around them?

Jäger (Reference Jäger2007) introduced a set of games—‘sim-max games’—that can help address this problem. Sim-max games are modified versions of the signaling game, introduced by Lewis (Reference Lewis1969), that assume actors are transferring information about states of the world that bear underlying similarity relations to one another. What this means is that, unlike in a traditional Lewis signaling game, there is a natural sense in which actors in sim-max games might want to categorize groups of states together in order to transfer meaning about them.Footnote ¹

This article will use sim-max games to model perceptual categorization, with the goal of answering a question related to the problem mentioned above: To what degree should we expect the perceptual categories of biological actors to track properties of the world around them? I will argue that an analysis of these games suggests that the relationship between real-world structure and evolved perceptual categories is mediated by successful action in the sense that organisms evolve to categorize together states of nature for which similar actions lead to similar results. This conclusion indicates that both strongly realist and strongly antirealist views about perceptual categories are too simple.

Before beginning, I would like to make a methodological remark. One might worry that the central observation of this article—that perceptual categories can be expected to group real-world items for which the same actions are effective—could be made without appeal to game theory. One can give an intuitive argument for this observation; evolution responds to payoff, and thus perceptual categories will evolve to track payoff rather than natural structure. And, in fact, both vision researchers and philosophers of color have made arguments along these lines.Footnote ² The game theoretic framework presented here, though, does two things that this intuitive argument does not. First, sim-max games provide mathematical justification for what might otherwise seem to be a hand-wavy conclusion. Second, the framework brings conceptual clarity to a topic that is complex and many faceted. While the thesis of the current article may seem obvious, or even trivial, from a strategic point of view, it is certainly not universally accepted by philosophers or scientists studying perception.Footnote ³

In section 2, I describe sim-max games. In section 3, I outline optimal strategies for these games and discuss their evolutionary properties. In section 4, I describe how these games can be used as a model of the evolution of perceptual categories and then proceed to make the principal arguments of the article.

2. Signaling Games and Sim-Max Games

The standard signaling game, as outlined by Lewis (Reference Lewis1969), is often taken as a model of information transfer between agents. The game has two players—a sender and a receiver—and three stages. In stage 1, exogenous forces, or ‘nature’, determine the state of the world. In stage 2, the sender observes this state and sends a signal to the receiver contingent on it. In stage 3, the receiver observes the sent signal and chooses an action. If the action is appropriate for the state of the world, both players receive a payoff. If the action is inappropriate, neither player receives a payoff. The goal of both players is to coordinate the action taken by the receiver with the state observed by the sender. Neither player cares how this coordination is achieved, that is, what specific signal is used to denote any particular state, as long as coordination occurs.

Sim-max games build on this model by adding similarity structure to the state space of the signaling game. In their basic form, these games model situations in which organisms would like to transfer information about properties that vary finely, or even continuously. In these cases, because there are many relevant states of the world that bear similarity relations to one another, categorization is potentially useful for transferring information. Properties that fit this description are things like distance, size, degree, concentration, color, time duration, temperature, and so on.

In sim-max games, states of the world are modeled as points in a metric space where distance represents similarity.Footnote ⁴ The greater the distance between two states, the less similar they are and vice versa. In these games, as in the Lewis signaling game, ‘nature’ selects a state of the world. The sender observes this state and sends a signal. The receiver observes this signal and chooses an act. If the act is perfectly appropriate for the state of the world, the actors receive identical, perfect payoffs. However, the key alteration to the game is that if the act is not perfectly appropriate, the actors still receive a payoff based on how nearly appropriate the act is. In other words, the game incorporates payoffs that vary as a function of the distance between the state of the world and the act taken. Specifically, payoff decreases as distance between the state of the world and the act taken increases. This aspect of the model mimics many real-world situations in which states bear similarity relations to one another. In these cases, because states are similar, the same action will often be appropriate (or nearly appropriate) for a number of them. If we are signaling about the ripeness of an apple, for example, I might group a number of apples with different intensities of redness under the signal ‘ripe’. If you bite into any one of these, you will receive a payoff because the apples are ripe or close to it. This will be the case even if some of the redder apples are slightly more ripe than your ideal apple, and some of the greener ones are slightly less so.

Before continuing, it will be useful to say a word about similarity, given the importance of this concept to the sim-max model. Similarity is a complex and much debated notion. In the most clear-cut cases, the claim that real-world states can be more or less similar is perhaps not contentious. Consider a situation in which the relevant states of the world correspond to the length of a bridge. It is not strange to say that a 200-foot bridge is more like a 201-foot bridge than a 4-foot bridge, holding all other features constant. In cases like this, one is considering similarity with respect to some feature of objects.Footnote ⁵ Moving away from these cases, one quickly runs into confusing situations. Is a banana objectively more like a skyscraper or a wink? For the purposes of this article, problems regarding similarity will have to be set aside. It will be assumed that there are ways in which real-world states are similar and dissimilar. The lessons drawn about relationships between perceptual categories and real-world similarity, though, should be taken to apply more clearly in less contentious cases of similarity.Footnote ⁶

3. Optimality, Equilibria, and Evolution

Having outlined sim-max games, it will now be useful to discuss these games in more detail in order to understand how they shed light on perceptual categorization. In particular, I will discuss optimality and some results from evolutionary game theory that will help elaborate what sorts of strategies should be expected to evolve when actors are playing sim-max games.

3.1. Strategies and Optimality in Sim-Max Games

Optimal strategies in sim-max games are those strategies in which similar states are categorized under the same signal and in which the actor then takes an action in response to that signal that is at least partially appropriate for all the states attached to it. In other words, the receiver is taking acts that are as appropriate as possible as much of the time as possible. Such a strategy will always garner the highest payoffs for the actors, due to the assumption that payoff decreases with distance. The question, then, is which strategies allow the actors to most perfectly coordinate action?

To understand these strategies, it will first be useful to describe what is called a Voronoi tessellation. A Voronoi tessellation is a division of a space around what are sometimes called ‘generators’ or ‘seeds’. These generators consist of locations in the space. The tessellation is a division of the surrounding space into cells, where every point is assigned to a cell on the basis of which generator it is closest to. Figure 1 shows four Voronoi tessellations. The first two are of a straight line. Each point pictured in figures 1a and 1b is a generator, and each cell of the tessellation is represented by two vertical dashes on either side of these generators. Note that in each case the vertical dashes are exactly equidistant from the generators on either side of them. Figures 1c and 1d show Voronoi tessellations of a two-dimensional space. Each generator is represented as a point, and each cell contains only those points in the space closest to the generator at the center of that cell.

Figure 1.

Voronoi tessellations of a line (a, b) and a two-dimensional space (c, d).

Suppose that the space in figure 1d represents the state space of a sim-max game. Furthermore, suppose that each point represents the ideal state for an act that a receiver is taking in response to some signal. The cells of the Voronoi tessellation represent the optimal response by the sender to the receiver. For any particular state of the world, the sender assigns that state a signal on the basis of which cell it is a part of. Given which acts the receiver is taking for each signal, this sender strategy ensures that each state will garner the highest payoff possible, because it will be matched with the closest possible act. Given this type of optimal sender strategy, receivers then optimize by spacing out signal interpretations so that the Voronoi cells associated with each one will be nearly the same size.Footnote ⁷ Figures 1b and 1c represent Voronoi strategies with this sort of optimal character.Footnote ⁸

4. Applying Sim-Max Games to Perceptual Categorization

Signaling games are most often taken as models of information transfer between organisms. In fact, though, signaling games can also be used to understand within-organism processes. In particular, signaling games can be interpreted as representing perceptual signaling. Under this interpretation, the state of the world is a type of real-world item that causes perceptual input, while the act taken is the reaction of the organism to this perceptual input. It is difficult to say exactly what, in this scenario, the signal is. However, if one allows for a broad interpretation, the signal can be taken to be a mediating mental process that starts with perceptual input and ends with behavior. In particular, one can take the perceptual experience of an organism to be an important part of this signal.Footnote ¹⁰

With this interpretation, one can use sim-max games as a model of perceptual categorization. Upon doing this, the first thing to note is that the evolved categories in sim-max games—Wagner’s results notwithstanding—are convex. A convex set in a metric space is one where for any two points in the set, every point on a straight line segment between these two points is in the set as well. The cells of Voronoi tessellations are necessarily convex, and so the categories that evolve in the long run in sim-max games are convex as well. Convexity is interesting here because it means that actors evolving perceptual categories should be expected to group together states of the world in ways that reflect what are arguably natural properties of these states. Like things will be grouped together, sufficiently unlike things will not. In other words, perceptual categories should be expected to track real-world structure in that similar real-world items will belong to the same category while sufficiently different real-world items will not.

However, a closer look at just how similarity is built into the structure of sim-max games indicates that, in fact, this conclusion is not warranted. In describing sim-max games, I made an appeal to the geometrical structure of the state spaces of the games. Throughout the article, to this point, distance in this space has been described as representing similarity between different states of the world. This is a natural way to describe these games and to give an intuitive sense of how they work and why they behave as they do. But, in fact, this intuitive way of understanding sim-max games is somewhat misleading. Similarity relations in the state space of sim-max games are built in through payoff structure alone. In other words, there is no necessary sense in which two states are like or unlike each other except in that the same acts will receive payoff for both states, or will not. The game can be described perfectly, without appealing to geometry, as a set of states, a set of acts, and a payoff defined for each pair of these. Another way to put this is as follows: from a modeling perspective, two states, even if they are objectively wildly different, should be treated as occupying the same spot in state space in a sim-max game if the same acts obtain the same payoffs in both states.

What this means is that results from sim-max games only indicate what follows: organisms will evolve to categorize together states of the world for which the same act is appropriate, whether or not these states are somehow similar or dissimilar. Organisms, then, should be expected to evolve perceptual categories not directly on the basis of how real-world properties relate but on the basis of how organism action either contributes or does not contribute to the organism’s success when these properties are present.Footnote ¹¹ Another way of saying this might be that perceptual categories should track payoff similarity rather than real-world similarity.

The observations above, at first glance, may seem to strike a significant blow to those committed to the idea that perceptual categories accurately track real-world properties. Sim-max games seem to indicate that there is no particular reason to think this is the case. Mark et al. (Reference Mark, Marion and Hoffman2010) have argued something similar on the basis of a different set of evolutionary game theoretic models. They have shown that in competitive choice situations, perceptual categorization strategies that do not accurately mimic real-world structure can outperform those that do. Along these lines, Hoffman claims that perceptual categories should not be assumed to accurately reflect real-world properties because, as he says, “Fitness, not accuracy, is the objective function optimized by evolution” (Reference Hoffman, Dickinson, Tarr, Leonardis and Schiele2009, 151).

Such considerations might lead one to a strong antirealist position with regard to perceptual categories—that there is no reason to think they track natural properties—and this is indeed what Hoffman defends.Footnote ¹² But sim-max games actually support only a more modest claim. According to the sim-max game framework, perceptual categorizations are determined through payoff constraints rather than directly by real-world properties. A strong antirealist position requires an additional assumption, however, which is that there is little relation between real-world structure and payoff structure. And this assumption is unwarranted.

It seems likely, for example, that when organisms encounter two highly similar real-world situations, the same actions will be appropriate to both. Take two scenarios that are identical with the exception that in the first a leopard is 20 feet away from a vervet and in the second the leopard is 21 feet away. Because the physical realities of the two scenarios are similar, it will be appropriate for the vervet to take the same action in both cases. In many cases, similar real-world states should, in sim-max games, be mapped to the same area of state space because they are appropriately paired with the same sets of actions. If so, it should then be expected that perceptual categories often track real-world similarity properly.

It is interesting to note, however, that in many situations, the same cannot be said about the differentiation of unlike states. It is often the case that similar actions will be appropriate for states of the world that are objectively different. If an escape pattern for vervets works both to avoid, say, leopards and human hunters, there is no reason not to use the same category to denote these two quite different groups of states. This argument about categorization is well embodied by the example of human color perception. It is notoriously difficult to identify some objective property of the world that color experience perfectly corresponds to. The most promising property is the surface spectral reflectance profile, or SSR, of a surface. This profile is a measurement of the probabilities with which wavelengths of light will reflect from a surface. While nearly identical SSRs will stimulate similar color experiences in observers, sometimes quite different SSRs will also stimulate similar color experiences in observers. In other words, color categorization respects similarity but not necessarily difference.

5. Conclusion

To conclude, I would like to reemphasize the two main insights that the strategic framework presented here can provide with regard to perceptual categories. (1) Perceptual categories should be expected to evolve such that states that can be responded to with the same actions are grouped together perceptually and states that cannot are not. (2) The fact that evolutionary processes are guided by payoff does not necessarily mean that perceptual categories will fail to track real-world structure. Neither, though, do these results support what Hoffman (Reference Hoffman, Dickinson, Tarr, Leonardis and Schiele2009) calls the ‘conventional view’ of perception, that “the primary goal of perceptual categorization is to recover, or estimate, the objective statistical structure of the physical world” (149). Instead, the sim-max game framework provides a better understanding of how perceptual categories should be expected to relate to real-world structure on the basis of how real-world structure influences organism payoff. This relationship means that perceptual categories should not necessarily be expected to either track or not track real-world structure but rather to bear more complicated relationships to it.