1. Introduction: The Underdetermination of Retinal Stimuli

In “An Essay Towards a New Theory of Vision” Reference Berkeley([1709] 1975), Berkeley posed a puzzle that theorists have had to grapple with ever since. Given the transformations that occur when sourcesFootnote ¹ in three-dimensional space project onto a two-dimensional surface, the retinal image cannot be used to determine the size, distance, and orientation of the real-world geometry that produced the stimulus, since the full dimensionality of the world is not preserved. This problem becomes only more intractable when it is recognized that the quality and quantity of light returned to the eye further intertwines the relative contributions of illumination, surface reflectance, and atmospheric transmittance, thereby making it impossible to recover how each of these factors have been combined to produce the patterns of light projected on the retina. As a result of such circumstances, retinal stimuli are underdetermined with respect to the world.

Stated in such terms, the fact that projected images are underdetermined might not seem to be a significant problem since we do not perceive the retinal image. But if retinal stimuli contain the only direct visual information an observer has about the world, yet such stimuli cannot uniquely specify the conditions of that world, then explaining how observers generate visual representations that are useful guides to the sources of stimuli presents a profound challenge (cf. Purves and Lotto Reference Purves and Lotto2003; Wojtach Reference Wojtach2005). This puzzle has come to be known as the “inverse optics problem” (Palmer Reference Palmer1999).Footnote ²

To better understand the geometric aspects of the inverse optics problem, consider the relationship between the projection of a straight line on an image plane (or retina) and the real-world conditions that could have generated the image (Figure 1). From the projected stimulus alone, it would be impossible to determine the three-dimensional source that is the unique cause of the projection, since any number of sources of different sizes, at different distances from the observer, and at different orientations in space could all subtend the same visual angle in the image. Notice that this problem remains regardless of whether the image is a static scene, as in Figure 1, or a dynamic scene, as in Figure 2.

Figure 1.

The inverse optics problem. Due to perspective transformation, objects of different sizes, at different distances, and in different orientations can all generate the same projection on an image plane. Therefore, a projected image cannot uniquely specify its source in three-dimensional space.

Figure 2.

The inverse optics problem as it pertains to moving objects. (A) Three objects of different sizes, at different distances, and in different orientations project the same image, as in Figure 1. (B) The same objects moving with different speeds in space can generate the same sequence of projections on the image plane (see also [C]). Similar to a static image, then, a moving image cannot uniquely specify its source in three-dimensional space.

If the sources of Figure 1 were moving in three-dimensional space from right to left (Figure 2, frames A to C), then the projected image sequence illustrates a further aspect of the inverse problem: in addition to the problems posed by size, distance, and orientation, the speed of a source cannot be uniquely determined with respect to a moving image on the retina (Wojtach et al. Reference Wojtach, Sung, Truong and Purves2008). Therefore, even though the series depicted in Figure 2 contains only three different sources moving in multiple frontal planes with various speeds, an infinite number of sources of different sizes, at different distances, with various orientations, and traveling at different speeds could have produced the stimulus. As a result, the inverse problem pertains to full range of speeds and geometries that can arise from three-dimensional space.

As mentioned above, the conflation of information also affects the quality and quantity of light projected onto the retina. Perhaps the easiest way to capture this aspect of the inverse problem is to consider achromatic stimuli—in particular, the puzzle introduced by simultaneous brightness contrast (Figure 3).Footnote ³ As is well known, luminance is an objective measure of the intensity (or quantity) of light returned to the eye from physical sources as measured by a photometer. We do not perceive luminance, however, but lightness and brightness, typically measured by having an observer indicate the appearance of one surface relative to another.Footnote ⁴ Intuitively, one might think that if two surfaces in a scene returned the same amount of light to the eye (i.e., had the same luminance), then those surfaces would be perceived as equally bright; as demonstrated by the example of simultaneous brightness contrast, however, this is not the case.

Figure 3.

Simultaneous brightness contrast effect. When two target surfaces returning the same physical amount of light to the eye (i.e., having equal luminance) are each surrounded by surfaces returning different physical amounts of light (i.e., having different luminances), the targets appear to be differently bright. In this case, the small target rectangle surrounded by the background of relatively low luminance on the left appears to be brighter than the small target rectangle surrounded by the background of relatively high luminance on the right. See text for the significance of this effect.

In Figure 3, two small target rectangles of equal luminance are each surrounded by larger backgrounds with different luminances. When the targets and their respective backgrounds are presented adjacent to one another, so that they can be compared, the targets appear to be differently bright: the target surface surrounded by the background of relatively low luminance on the left appears to be brighter than the target surface surrounded by the background of relatively high luminance on the right. Thus, simultaneous brightness contrast is a simple means to illustrate an important point: our visual experience is not a veridical representation of either the retinal image or the underlying reality of sources in the world.

While the physiological details concerning how the visual system generates perceptions of brightness remain enigmatic, the underlying rationale for why it does so can be offered by taking into account the circumstances posed by the inverse problem. For unless the visual system had some way of “reverse engineering” luminance values to reveal the actual conditions of illumination, surface reflectance, and atmospheric transmittance that together generated the stimulus, relying on luminance alone (or any property of the retinal stimulus as such) to determine the content of visual experience would be a poor method for representing the world and guiding behavior. As described below and in Section 4, there are good reasons to believe that a better tactic—one that results in perceptions of brightness, not luminance—evolved to contend with such circumstances.

Whereas Berkeley argued that the inverse problem could be overcome by coupling visual and tactile information, Helmholtz hypothesized that past experience might be relied upon to supplement the information in the current retinal stimulus (Helmholtz Reference Knill and Richards[1866] 1924). Described as a process of “unconscious inference,” the use of such experience could generate perceptions more appropriate to the sources that generated the retinal stimulus than might be achieved by using the properties of the stimulus alone. More recently, this general idea that the visual system must in some way use past experience broadly construedFootnote ⁵ to inform perception and behavior has been employed to great advantage with what have come to be called “probabilistic” models, thereby formalizing Helmholtz's proposal of how the brain might effectively contend with the inverse optics problem.Footnote ⁶

The underlying framework common to all probabilistic models is that evolution by natural selection has shaped the design of perceptual systems in response to environmental conditions. Because of this, the statistical regularities of natural environments can be used to investigate how organisms have come to represent the world (Simoncelli and Olshausen Reference Simoncelli and Olshausen2001). When adopting a probabilistic approach, however, the chosen method can impact how one understands visual representation and the determination of content. For example, some probabilistic models apply Bayes’ theorem to compute the probability distributions of real-world states that could have given rise to a retinal stimulus (cf. Knill and Richards Reference Helmholtz1996; Maloney Reference Maloney, Heyer and Mausfeld2001; Rao, Olshausen, and Lewicki Reference Rao, Olshausen and Lewicki2002; Weiss, Simoncelli, and Adelson Reference Weiss, Simoncelli and Adelson2002; Stocker and Simoncelli Reference Stocker and Simoncelli2006). While the perceptual “decision” resulting from this technique can be expressed in various ways (e.g., as a maximum a posteriori, or MAP, rule for selecting the mode of the posterior distribution), Bayesian models tend to assume that a visual representation corresponds to the most likely physical source of a stimulus. Due to the constraints posed by the inverse optics problem, however, this may not be the best way to conceive of representation.

Another probabilistic model takes a different approach. Given that our visual experience does not conform to the physical properties of the retinal stimulus or the underlying source that produced the stimulus, representation can be conceived in terms that do not correspond to either. On this view, called empirical ranking theory, representations are not generated according to what the most likely properties of the retinal stimulus or real-world source might be, but by the relative rank a given retinal stimulus has in relation to all other instances of the same (or similar) stimulus parameter in accumulated past experience (Yang and Purves Reference Yang and Purves2004; Howe and Purves Reference Howe and Purves2005; Wojtach Reference Wojtach2005; Long, Yang, and Purves Reference Long, Yang and Purves2006, Wojtach et al. Reference Wojtach, Sung, Truong and Purves2008). As a result, visual representations are taken to be empirical constructs that cannot be directly mapped onto reality.

To better understand this position, consider the puzzle of simultaneous brightness contrast illustrated in Figure 3. Instead of attempting to explain the perception of brightness in terms of the luminance values of the stimulus or the most probable surface reflectances of the source, the representation of differential brightness elicited from the equiluminant target stimuli is determined by how often such targets co-occurred with the full range of surrounding luminance values in natural environments. This strategy therefore relies on more than the luminance values of the target and surround in the present stimulus; instead, it depends on the entire distribution of conjoint luminance values in accumulated past experience. When these conjoint values of target and surround are ordered on a scale ranging from the lowest to the highest luminance according to their frequency of occurrence, a procedure called “histogram equalization” in information theory (Laughlin Reference Laughlin1981), the same target stimulus will have a different relative rank (corresponding to a different perceived brightness) depending on the surrounding context. As a result, the representation of brightness generated from the quantity of light falling on the retina can differ markedly, giving rise to the differences between perceived brightness and luminance (or any other visual quality and the measured properties of the stimulus) that are commonly noticed in the case of “visual illusions.” If this framework is right, then contending with the underdetermination of retinal images by relying on the relative rank of a given stimulus in accumulated past experience is the key to understanding representation and the determination of content.Footnote ⁷

2. Teleological Theories and Externalism

One immediate implication of the inverse optics problem is that internalism—the position that the content of visual states can be characterized with reference to internal factors aloneFootnote ⁸—must be incorrect, since external states of affairs will be essential in determining any visual representation. For as indicated by the inverse problem itself, there must be some way to link underdetermined images with external sources in order to generate useful percepts of the world. Although internal states are a necessary condition for visual perception, they cannot be sufficient to account for representation or the determination of visual content.Footnote ⁹ This seems to leave externalism as the only viable candidate for an explanation of visual representation; therefore, further clarifying remarks about internalism will be set aside.Footnote ¹⁰

To explain visual representation, two topics are of central concern: misrepresentation and the determination of content. Of the various externalist accounts that have been proposed, the naturalized perspectives offered by Dretske (Reference Dretske1981, Reference Dretske and Bogdan1986, Reference Dretske1995), Millikan (Reference Millikan1984, Reference Millikan1993), Fodor (Reference Fodor1987), and others under the rubric “teleological semantics” constitute perhaps the most significant attempts to address these problems. Despite a number of insights from this perspective, however, the underlying reason why these efforts have been unsatisfactory is because the inverse optics problem and its consequences are largely obscured. For as argued below, once the inverse problem is appreciated, the very idea of a “correct” and “incorrect” representation, as well as how content is determined, must be reconceived. In an effort to demonstrate this, particular attention will be paid to Dretske's teleological framework (Reference Dretske1981, Reference Dretske and Bogdan1986, Reference Dretske1995), since his view most easily illustrates these matters; it could be contended, however, that similar problems arise on other teleological accounts as well.

Dretske ultimately secures the ability of something to represent in the capacity of what he calls “natural signs” to indicate: shadows to the east mean (indicate) that the Sun is in the west; expanding metal means (indicates) that the temperature is rising (Dretske Reference Dretske and Bogdan1986). These signs mean what they do whether or not anyone or anything realizes it, and they can gain their meaning in two ways. One way is through lawful relations between the sign (or the sign's having a certain property) and the condition that constitutes its meaning—e.g., a shadow to the east (the sign) and the Sun being in the west (the condition). Another way to achieve the connection between natural signs and their significance is through nonlawful regularities—i.e., conditions that reflect regularities, but do so without the objective constraints of a law. Dretske's preferred example is that of a ringing doorbell (the sign) and the meaning it carries that someone is at the door (the sign's significance). While not a lawful relation, a regularity has been established such that when the bell rings this means (indicates) that a person, and not something else, is at the door. As long as this regularity persists, a ringing doorbell retains what Dretske calls its “natural meaning” (Dretske Reference Dretske and Bogdan1986, 19–20).Footnote ¹¹

An important limitation of natural signs, however, is that while they can indicate (and thus fail to indicate) something about the world, they cannot falsely indicate anything. If the Sun is not in the west, then shadows to the east do not retain their natural meaning that the Sun is in the west; similarly, the ringing doorbell cannot be a natural sign that someone is at the door when no one is there (Dretske Reference Dretske and Bogdan1986, 20–21). So construed, natural signs cannot be said to represent their conditions, since they cannot offer the ability to misrepresent. To explain this ability—and therefore to explain “genuine” representation—Dretske argues that a connection must be made between the functional roles of systems and natural signs. If a system (or component of a system) has an identifiable function, then there is a way to speak of that system (or component) as misrepresenting the conditions it has otherwise been designed to indicate; this is the system's “functionally derived meaning” (Dretske Reference Dretske and Bogdan1986, 22).

To understand Dretske's aim, consider the more problematic case of the ringing doorbell. In normal circumstances, the doorbell system naturally means (indicates) that someone is at the door. Yet the doorbell, as a designed system with an identifiable function, still means something in abnormal circumstances; when it is malfunctioning, it can mean that someone is at the door when in fact no one is there. Thus, according to Dretske, the doorbell system can functionally mean (represent) something about the conditions it was designed to track because under normal circumstances it has natural meaning (indication).Footnote ¹² For Dretske, such conditions are constitutive of misrepresentation and, therefore, representation.

This same general point can be made for biological systems: if it is the natural function of a system (or component) to normally indicate the conditions it has evolved to signify, then under abnormal conditions, when the system fails to indicate something about the conditions it has evolved to signify, the system still retains a functionally derived meaning because it was supposed to naturally mean what it evolved to indicate. If this is correct, then we seem to have a teleological basis of the way content (and hence false content) might be instantiated in such systems (Dretske Reference Dretske and Bogdan1986, 22–25).

While a teleological approach seems to be a significant advancement in the effort to explain (mis)representation, the problem of how such representational content is determined remains. A particular strength of Dretske's position is that it offers a way to contend with the indeterminacy of content.Footnote ¹³ The problem here is whether the representational content of a visual state is of an F in the world, or of an F-or-G-or-H-or-. By relying on teleological function this problem is apparently overcome, since the content of a representational state is now determined in terms of a system's natural function—what a system (or component) is supposed to indicate. Therefore, if it is the natural function of a system to indicate an F, rather than a G or an H, then that is what the system naturally (hence functionally) means; that is what the system represents. The question is whether such an appeal to teleological considerations can guarantee this result.Footnote ¹⁴

3. The Importance of the Inverse Optics Problem

Teleological frameworks like Dretske's gain leverage on the problem of representation because they assume that since the function of simple systems can be straightforwardly linked to natural signs, the same holds for the function of complex systems. For reasons that stem from the inverse optics problem, however, this otherwise plausible assumption does not hold in the case of vision. First, consider that while Dretske's concept of a natural sign is unimpeachable, this concept only succeeds because signs are linked to their significance through lawful relations or nonlawful regularities in a one-to-one manner, as when a shadow to the east indicates that the Sun is in the west (as opposed to a light bulb to the north). Notice, however, that in the context of perception and the inverse optics problem, natural signs that work in one direction (from source to projected image)Footnote ¹⁵ do not work in the other: an image (the sign) cannot indicate the specific condition in the world that is its source (the sign's significance), since the mapping in this direction is a one-to-many relation (cf. Figures 1 and 2). This is problematic if visual representation is to be explained by connecting the functional role of the system to natural signs: even if the visual system has the identifiable function of visually representing the source of a projected image, the inability to link this function with a natural sign undermines the basis for a teleological explanation of representation.

Suppose, however, that despite these difficulties sense could be made of a natural sign in the context of the inverse optics problem. The obstacles to a teleological framework of visual representation still remain, since the link between indication and function is achieved by appealing to “normal” and “abnormal” circumstances. For once the idea that a system has evolved to represent external conditions is in place, it is a small step to the position that under normal conditions it has evolved to represent those conditions correctly. With this move now made, a natural corollary is to maintain that when normal conditions in the environment fail, so does the representation. Misrepresentation, then, comes to be identified with the states of systems that are in error because they do not correctly represent the external conditions on which the functionality of the system is predicated. Thus, the implication of a teleological approach is that the functional system in question either operates properly, as it was designed to do, or it is malfunctioning; it is either representing correctly, or it is not.Footnote ¹⁶

To make this point clear, consider the following. A teleological framework like Dretske's assumes that because the visual system

1. 1. evolved to (is supposed to) represent external conditions,

the functionally derived meaning of the system can

1. 2. represent those conditions as they really are.

But 2 contains a further commitment, namely,

1. 3. what the visual system represents is what we see.Footnote ¹⁷

There is much to be said for the position that a biological system's natural function is to represent its environment, as captured in statement 1. One obvious benefit is that now evolutionary history can be used to define the representational parameters of the system (i.e., what it represents), rather than having to consider the visual system's present state and proceed from those facts alone (cf. Millikan Reference Millikan1993). But while such reasoning seems to apply unproblematically to statements like 1, this general notion of representation cannot be extended to include the phenomenology of visual representations, as expressed in statements 2 and 3: for given the relationship between an image and its source as described by the inverse problem, the ability to characterize the representational product of vision as “correct” or “incorrect” must be questioned.

This is a contentious claim. It might be argued, therefore, that while the inverse problem poses a challenge for the way organisms visually represent the world, it does not follow that the very notion of a correct (or incorrect) representation as expressed in a teleological framework is called into question. For suppose that all the visual system can do is contend with (rather than solve) the inverse problem. When it contends with this problem successfully, by generating a representation of the real-world source of the stimulus, then this is all that is required to capture what is at issue with a correct representation; conversely, when a representation is not of the source that generated the stimulus, this would describe an incorrect representation. Could not a teleological approach still operate with these modified notions of correctness and incorrectness in place, and thereby explain (mis)representation with respect to vision?

While this reasoning seems to address the issue, it fails to notice some important consequences ushered in by this shift in the meaning of terms. First, by using ‘correct’ to now mean “contend with successfully,” the term is no longer applied in its original sense. Whereas before it was meant to convey something like veridicality, or representing the world in accord with normal conditions, it now carries a meaning far from the manner in which it was initially employed. A “correct” representation in this revised sense—i.e., where the term refers to some criterion of accessibility for representational states—is not what Dretske (and others) had in mind when offering a teleological explanation of (mis)representation or the determination of content.

Suppose, however, that something like this modified notion of a correct representation were the aim of a teleological position, the objective being to secure the conditions for assessing a given representation. Notice that if this were the case, the result is no better. For regardless of how “assessability” is defined in this context, such a position would have to maintain that the source of a stimulus could be represented, since only then could the representation be assessed. Indeed, a teleological approach must maintain this, given that the foremost strength of this perspective is the professed ability to explain the difference between a representation and a misrepresentation. As the inverse problem demonstrates, however, visually representing the source of a stimulus cannot be accomplished. Therefore, even allowing a representation to be couched in such terms cannot salvage a teleological approach. Without a way to represent (and thereby assess) the source, the defining characteristic of a (mis)representation no longer remains.

To evaluate these conclusions in terms of a concrete example, consider the simultaneous brightness contrast effect of Figure 3. As argued above, a correct (i.e., veridical) representation can be easily dismissed, since this construal would require that the luminance values of the stimulus could be analyzed to reveal the actual contributions of illumination, surface reflectance, and atmospheric transmittance that comprise the stimulus. Because this cannot be accomplished (cf. Section 1), there is no way to achieve a veridical representation of the scene. For similar reasons, a representation was also shown not to be assessable for correctness on a teleological approach, since the strong dichotomy between a representation and a misrepresentation required that the source could be represented.

Despite these arguments, it may still seem as though some notion of correctness is appropriate to ground the difference between a representation and a misrepresentation from a teleological perspective. One might therefore maintain that effects like those elicited by simultaneous brightness contrast are epiphenomenal in their character, an illusion or misrepresentation that occurs only in isolated circumstances and without any real behavioral efficacy. According to this view, in the vast majority of conditions the visual system represents the world correctly (in some sense of the term); only in a few instances—broadly cast under the net of “visual illusions” or “misrepresentations”—does the visual system fail.

The problem with this notion of “correct,” however, is that it requires an explanation of our visual experience in the preponderance of cases. If it cannot be the actual conditions that generated the stimulus, then presumably it must be the properties of the stimulus itself—e.g., the luminance values projected on the retina. This might seem to be an adequate resting place, but it quickly generates problems of its own. One problem is that it now seems that what the visual system represents is the retinal stimulus—a position few, if anyone, would find attractive. A related problem is that it now makes the simultaneous brightness contrast effect even more perplexing. For since the visual system would now have to behave like a photometer by representing luminance, an explanation for why the system fails in the case of simultaneous brightness contrast must be offered.Footnote ¹⁸ And if the culprit has something to do with perceiving in terms of brightness rather than luminance, then this would seem to relegate all cases where brightness is perceived to the class of misrepresentations—in effect making all visual representations misrepresentations, since brightness perception is a fundamental aspect of vision. Unless one simply denies the simultaneous brightness contrast effect—a desperate move that merits little attention—there seems to be no work remaining for a correct representation to do.

The crux of the matter, then, is that for a teleological framework like Dretske's to work it must be possible to delineate between correct and incorrect representations, since that is how functionally derived meanings explain the problem of representation. But since the inverse optics problem rules out this representational dichotomy, such a framework cannot offer either a naturalized account of visual content or an externalist justification for determining the content of visual states. In a very real sense, then, by moving the inverse problem to center stage (rather than ignoring or obscuring it) a different approach is required to explain visual representation. This is an important consequence, since it appears that an internalist position is also flawed in this regard (although for different reasons). The question then becomes, how can these two issues—naturalizing visual representation and content determination—be explained?

4. Teleosemantics Revisited

Dretske's teleological framework has been highlighted in part because it offered the most direct means for reaching the heart of the matter concerning visual (mis)representation. In this sense, it has been used primarily as a vehicle for establishing a negative conclusion. But this was not the only reason for focusing on Dretske's view; there is much in his argument that can be put to good use, despite the problems inherent in such an approach. The first step in preserving aspects of a teleological framework, then, would be to establish some of the more important merits of the position.

Perhaps the best reason has already been stated: using evolutionary history to define the representational parameters of the system (i.e., what it represents), avoids the intractable task of having to define visual representation in terms of the system's present state alone. Given the inverse optics problem, and the complexity of the task the visual system must contend with, appealing to evolved function is perhaps the only option. Evolved function by itself, however, cannot describe how the contents of visual perception are actually determined; for this, a perspective such as empirical ranking theory must be adopted (see Section 1).

For visually guided creatures, survival demands that complex spatiotemporal retinal stimuli are processed at rapid speed. That these patterns of light cannot uniquely determine the underlying structure of the world makes the representational capacities of the system remarkable, but not inexplicable. By generating visual representations according to how often a given retinal stimulus has occurred relative to all other instances of the same stimulus parameter in accumulated past experience, however, the efficiency by which the brain is able to represent the world can be understood.

On this view, the success achieved by the visual system arises by linking underdetermined retinal stimuli to their real-world sources empirically, on the basis of behavioral feedback accumulated over evolution and development. By instantiating such information over time in the circuitry of the system itself, vast amounts of trial and error would tend to accrue in the system, thereby providing the computational basis for representation and the determination of content in the context of the inverse problem.Footnote ¹⁹ As stated in Section 1, the evidence for this strategy comes from analyzing large natural scene databases, which serve to approximate the relationships between projected images and real-world sources that humans and other visual animals would have extracted behaviorally over time (Yang and Purves Reference Yang and Purves2004; Howe and Purves Reference Howe and Purves2005; Long, Yang, and Purves Reference Long, Yang and Purves2006, Wojtach et al. Reference Wojtach, Sung, Truong and Purves2008). In adopting this tactic, however, a visual system could only come to generate useful—but not correct—representations. And although such representational states would be assessable, they are only assessable in terms of their biological usefulness to the organism.

If this is right, then the visual system can still be said to have evolved to represent the external world as in statement 1 above, much as Dretske argued for. But this fact alone does not offer any insight into the perceptual capabilities of the system as indicated in statements 2 and 3. The reason this was not an issue before was because one general notion of representation predicated on notions like natural signs and normal/abnormal conditions was deemed adequate for explaining visual representation and the determination of content. Thus, the real importance of the inverse problem is it demonstrates that if progress is to be made on the problem of visual representation and representational content, looking to what biological systems are supposed to represent (as expressed in 1 above) is not enough. In addition to this, the manner by which systems represent external conditions must be elucidated, since the contents of perceptual experience, as expressed in 2 and 3, will only be understood once this is done.

Consider once more the case of simultaneous brightness contrast presented in Figure 3. As previously stated, if the visual system could unravel the actual contributions that illumination, surface reflectance, and atmospheric transmittance had in generating the projected image, then the luminance values of the stimulus could be used to represent the world. But since this task cannot be accomplished—this just is the puzzle posed by the inverse problem—the visual system must be using a different strategy to create its representations. The approach of empirical ranking offers an explanation of how this might be achieved; it also points to a naturalistic explanation for the brightnesses (or any other subjective visual quality) that we experience on a routine basis.

As it pertains to Figure 3, then, we represent the scene in terms of brightness (not luminance) because the best way to produce biologically useful representations from underdetermined stimuli is with respect to the rank a particular stimulus has relative to the full range of conjoint luminance values that have occurred in the past, rather than by the actual luminance values on the retina at the present. The simultaneous brightness contrast effect arises, then, not because the system is failing to represent such circumstances correctly, but because an empirical strategy is relied upon to contend with the inverse problem. Although counterintuitive, this strategy can therefore explain why the contents of perception do not align with either the retinal stimulus or with the real-world conditions that generated the stimulus. In so doing, it provides a rationale for the discrepancies between appearance and reality that are most easily noticed in the case of the simultaneous brightness contrast effect and other so-called visual illusions.

This is also why it no longer makes sense to separate visual representations into two classes—those that are “correct” and those that are “incorrect” or “illusory”; instead all visual percepts are generated using the same empirical strategy (Purves and Lotto Reference Purves and Lotto2003; Wojtach Reference Wojtach2005; Wojtach et al. Reference Wojtach, Sung, Truong and Purves2008). It is not the case, then, that the visual system relies on this approach in some circumstances but not others; rather, the same framework is employed ubiquitously. The fact that this strategy cannot reveal reality (even though it may seem otherwise) only provides further support for the conclusion that the very ideas of correct representations and misrepresentations no longer apply.Footnote ²⁰

When conceived in these terms, it should be clear that a teleological perspective provides a way to describe what the visual system is representing—i.e., the sources in the world that generate visual stimuli. This is what the visual system was selected for; this is what its representations are supposed to be about. But such a perspective does not tell us much about how the system might actually accomplish this feat; moreover, it cannot predict—qualitatively or quantitatively—the perception elicited by a given stimulus, whereas a probabilistic framework like empirical ranking theory can (Purves and Lotto Reference Purves and Lotto2003; Wojtach Reference Wojtach2005; Wojtach et al. Reference Wojtach, Sung, Truong and Purves2008). If a theory of (visual) representation is measured by how well it can account for the relevant phenomena, then an approach that relies on past experience to link stimuli with sources by way of behavior enjoys much greater success.

Such considerations illustrate why a teleological perspective is inadequate, while offering a way to address how subjective experience can arise in a naturalistic framework. In particular, empirical ranking theory provides a way to recast 3 as

1. 3*. what the visual system represents empirically is what we see.Footnote ²¹

In addition, this manner of conceiving of visual representation also permits a way to recast statement 2. For now it can be stated that, because a biological system like the visual system

1. 1. evolved to (is supposed to) represent external conditions,

the system can

1. 2*. represent those conditions empirically, in terms of their biological utility.

When visual representation is thought about in these terms, it is a stark reminder that a teleological framework and the notions of (proper) function and malfunction, correctness and error, are poor descriptions of what is actually taking place. Such notions should instead be replaced with a probabilistic framework like empirical ranking theory and the corresponding idea of biological usefulness, since doing so avoids the problems that are raised when conceiving of vision in terms of “correct” and “incorrect” representations. Notice, too, that just because this strategy lacks the notion of a “correct” representation, it would be wrong to assume that it lacks epistemic status. If a visual system is relying on the relative ranking of stimuli in past experience to represent the world, then this is an epistemic result if there ever was one. Just because it is a different epistemic result than expected (i.e., not one grounded on direct access to the world) is no argument against this model of representation.

5. Empirical Externalism

If progress is to be made on the issue of (naturalized) perceptual content, it should be clear that a teleological framework can ground a broad notion of representation. To explain the manner by which we perceive the world in the context of the inverse optics problem, however, more must be offered than just an appeal to teleological considerations. As might be expected, this shift in perspective has implications for an externalist position regarding content.

A standard reading of externalism maintains that the properties of a representational state are fixed by external factors. In the context of vision, this means that real-world sources are represented as having particular shapes, colors, and the like because those sources actually have such properties. What the inverse problem so vividly illustrates, however, is that this manner of thinking about visual representation cannot be correct. For while the relationship between real-world sources and their projected images are to be located externally, the underdetermination of retinal stimuli precludes any way of representing the true properties of those sources. The position offered here has argued that to contend with such circumstances, the visual system employs a probabilistic strategy based on the empirical rank of a stimulus to represent the world; but in so doing, the ability to represent the world correctly is traded for the ability to represent the world usefully. If this is accurate, then the content of a representational state cannot be fixed by a particular image–source relationship (cf. Figures 1–3); only accrued information from the past could do this. Therefore, because this position is significantly different from traditional views on representational content, yet it retains at least some of the virtues of a teleological perspective, let us call this view of representational content empirical externalism. This term indicates what is correct about externalism—the connection between sources and their projected images is based on relationships that occur external to the organism—while signaling that, in the context of the inverse optics problem, it is the accumulation of images over vast amount of trial-and-error experience that determines content. Without taking such factors into consideration, a naturalized explanation of the representational content of vision would be beyond our reach. This shift of perspective therefore provides a different way to conceive of the content of visual states, while avoiding the standard problems inherent in the internalism/externalism debate.

While this solution indicates how representational content can arise, a corollary of this argument is that the traditional problem of indeterminacy remains: although the content of representational states are fixed by accumulated experience, given the inverse optics problem there is simply no way to uniquely determine the content of the system's representational states as they pertain to the world. The best that can be achieved is the determination of content probabilistically in terms of a stimulus’ empirical rank. This does not mean, however, that the way we represent the world is by chance: visual representations themselves are nonprobabilistic in their character, although they stem from a probabilistic system (cf. Wojtach Reference Wojtach2005). What it does mean is that, at least with regard to vision, the problem of indeterminacy is a red herring, since there is no solution that can be offered. Rather than worrying about this result, however, it is time to simply accept it. For given the constraints of the inverse problem, as long as a visual system permits an organism to generate biologically useful representations of its environment by using past experience, then this is all that is needed to characterize the content of such states.Footnote ²²

Unlike other philosophical perspectives on representational content, a particular strength of the position offered here is that it is testable: if the empirical rank of projected images from natural scenes cannot predict visual experience, then this hypothesis about representation and representational content would be demonstrably false. In fact, however, empirical ranking theory can predict what observers perceive with a high degree of accuracy (Wojtach Reference Wojtach2005), indicating the merit of this approach. Therefore, this argument supplies an extra dimension to the philosophical debate over representation by providing a specific, empirically motivated framework for how visual content can be realized.

Of equal importance, however, is the rationale for this stance: a probabilistic strategy like empirical ranking theory can explain the puzzling features of visual experience—the so-called “visual illusions”—that other frameworks relegate to the status of mere anomalies. Indeed, while such perceptions seem to be maladaptive, adopting the change of perspective argued for here provides a way to consider such “anomalous” perceptions differently—namely as hallmarks (rather than limitations) of the visual system's ability to represent the world. In other words, the alleged misrepresentations of vision are better understood as the more obvious manifestations of an empirical strategy.

In sum, once the inverse optics problem is recognized and taken seriously, new challenges for theories of visual representation are uncovered, and old concerns can be dispelled. For when viewed from the perspective presented here, it becomes clear that assigning content to a visual representation cannot be achieved by an appeal to teleological function alone; nor can it be had by looking simply to the external properties of the world or the internal properties of the system. Rather, such content is fixed by the way the system has been shaped via past experience so as to generate biologically useful representations. So understood, the result is a new framework from which to consider the problem of perceptual content.