1 Introduction

David Marr's Vision (Reference Marr1982) advances a computational approach to the study of visual processes. Although published almost 3 decades ago, its impact is still found in artificial intelligence, cognitive science, and neuroscience.Footnote ¹ Marr's work has also stimulated debates in philosophy about levels of explanation, top-down versus bottom-up methodologies, the nature of computation, externalism versus internalism in computational theories of mind, and the relations between content and computation.

Marr's conception of computational theory is somewhat unusual. In most computational approaches, the goal of a computational theory is to characterize the what and the how. The What element characterizes the function that is being computed, and the How element specifies the algorithm by which the system computes this function. But, according to Marr, this How element belongs not to the computational level but to “the algorithmic level.” In contrast, the two elements of Marr's computational theory are the what and the why: “The most abstract is the level of what the device does and why” (Reference Marr1982, 22).

But what exactly does Marr mean by “computational-level theory”? Some interpreters of Marr argue that computational-level theories specify the mathematical input-output function that is being computed (the algorithmic level then specifies the algorithm by which the system computes this function; see Egan Reference Egan1992, Reference Egan1995, Reference Egan2010; Butler Reference Butler1998). Other interpreters argue that computational theories also refer, in one way or another, to the visual content of these input and output states. This latter interpretation motivates the linkage between Marr's computational level and the “intentional” (Dennett Reference Dennett, Prawitz, Skyrms and Westerståhl1994), “semantic” (Pylyshyn Reference Pylyshyn1984), “knowledge” (Newell Reference Newell1982), or “ecological” (SterelnyReference Sterelny1990) level (see Sterelny Reference Sterelny1990, 44–46; Harnish Reference Harnish2002, 400ff.).

In what follows, I advance a different interpretation of Marr's conception of computational-level theories. This interpretation underscores the distinct character of the Why element in Marr's computational theories, an element that is often ignored.Footnote ² According to my understanding, Marr's computational level can be seen as intentional, in the sense that it is concerned with the embedment of the visual system in the environment we happen to live in. But it is not intentional if “intentional” boils down to the specification of visual content. Computational theories, I argue, relate the mathematical function that is being computed (specified by the What element) to the mathematical relations between the things that are being represented (specified by the Why element).

This article proceeds as follows: in section 2, I review some of the well-known statements of Marr. I suggest that one role of the computational level—the role of the What element—is to provide a function-theoretic, mathematical description of what the system does. In section 3, I argue that there must be another crucial component in computational-level theories, namely, the Why element. In section 4, I clarify the goal of the Why element, which is to explain why the mathematical function that is being computed is appropriate for a given information-processing visual task. In section 5, I argue that this goal is accomplished when it is shown that the “inner” input-output functional relations between the representing cells are similar (in ways specified in the article) to the mathematical relations between the entities that these cells represent. In section 6, I show how the Why element combines with the What element to constitute a computational-level explanation. In section 7, I compare my view with other interpretations of Marr.

2 Marr's Computational Theories: The What

In Vision, Marr (Reference Marr1982) famously advances a three-level approach to the study of visual processes. The computational level specifies what is being computed and why. The algorithmic level characterizes the system of representations that is being used (e.g., decimal vs. binary) and the algorithm for the transformation from input to output. The implementation level specifies how the representations and algorithm are physically realized.

According to Marr, neuroscientists have been occupied with the working of vision at the implementation level, namely, with the behavior and properties of cells that implement the visual processes and states. This approach, Marr argues, is descriptive and does not explain behavior.Footnote ³ The algorithmic level, which was imported to the study of vision from computer science, advances our theoretical understanding of the structure of visual processes. But it too does not capture essential elements of the task as an information-processing problem.

Marr thus concludes that “there must exist an additional level of understanding at which the character of the information-processing tasks carried out during perception are analyzed and understood in a way that is independent of the particular mechanisms and structures that implement them in our heads. This was what was missing—the analysis of the problem as an information processing task” (Reference Marr1982, 19). Marr emphasizes, here and elsewhere, two points about the study of vision. One is that the study of vision is primarily a study of an information-processing task: “Vision is therefore, first and foremost, an information-processing task” (3). The other is the prime role of the computational level in the analysis of visual, as information-processing, tasks: “Although algorithms and mechanisms are empirically more accessible, it is the top level, the level of computational theory, which is critically important from an information-processing point of view” (27).

What is an information-processing task? Marr characterizes it as “mapping from one kind of information to another” (24) or as “mapping from one representation to another” (31). He does not provide a detailed account of what is meant by representational content or information. Like many researchers of vision, Marr apparently identifies these terms with some sort of selective response to stimuli or with some reliable causal correlation between the activity of cells and certain types of stimuli. For example, he identifies “the apocryphal grandmother cell” (Reference Marr1982, 15) as “a cell that fires only when grandmother comes into view” (note on 15).

Characterizing an information-processing task somewhat schematically, we can present it as a process from one brain state (B₁) to another (B₂), whereby B₁ and B₂ are themselves representations. State B₁ represents objects/properties W₁, usually in the visual scene, and B₂ represents objects/properties W₂ (fig. 1). Examples from the context of vision are shape from shading, depth from disparity, structure from apparent motion, surface orientation from optical flow, and edges from light intensities (edge detection).

Figure 1.

Information-processing task that consists in a causal process (solid line) from brain state/event B₁ to brain state/event B₂, whereby B₁ represents (dashed line) an object/property W₁, in the visual scene, and B₂ represents object/property W₂.

Let us consider edge detection, the task of detecting oriented edges in the visual scene (fig. 2). State B₁, here, is the electrical activity of the photoreceptors, known as intensity values. This activity is sensitive to light intensities in the visual field, which consist, mainly, in light reflectance, geometry, illumination of the scene, and the viewpoint (W₁). These intensity values constitute what is known as the retinal image, wherein each “pixel” is sensitive to the light intensity in a certain location in the visual scene. State B₂ is the activity of cells in primary visual cortex (V1); these are roughly the Hubel and Wiesel's cells that are sensitive to oriented lines. These “edges” are salient features in the physical scene, such as object boundaries (W₂).

Figure 2.

Edge detection as an information-processing task. Intensity values of the photoreceptors represent light intensities in the visual field, which consists, among other things, in light reflectance. Cells in the primary visual cortex (V1) represent oriented edges, such as object boundaries.

What is the job of computational theories in analyzing an information-processing task? One job is to specify the functional relations between abstract properties of B₁ and B₂. Ideally, this mapping relation is defined in terms of mathematical properties abstracted from the electrical activity of the cells: “At one extreme, the top level, is the abstract computational theory of the device, in which the performance of the device is characterized as a mapping from one kind of information to another, the abstract properties of this mapping are defined precisely” (Marr Reference Marr1982, 24).

In their “Theory of Edge Detection,” Marr and Hildreth (Reference Marr and Hildreth1980; see also Marr Reference Marr1982, chap. 2) describe the early visual task of edge detection in terms of colocated zero crossings of different size filters of the form ∇²G*I(x, y) (fig. 3). The term I(x, y) refers to an array of intensity values of the photoreceptors, in which each pixel covers a spatial point (or region) in the visual scene. The term ∇²G*I describes the activity of retinal ganglion and Lateral geniculate nucleus cells, which perform a certain filtering of the intensity values. Here “*” is a convolution operator, and ∇²G is a filtering operator: G is a Gaussian that blurs the image, and ∇² is the Laplacian operator (∂²/∂x ² + ∂²/∂y ²) that is sensitive to sudden intensity changes in the image. The zero crossings of this formula are precisely those places in the image that have sharp intensity changes. This process takes place through several filters with different Gaussian distributions, each producing a different set of zero crossings (fig. 4). Detecting these zero crossings is the task of cells in the primary visual cortex (V1). The colocated zero crossings often signify edges, such as object boundaries, and are the basis for the zero-crossing (edge) segments in the raw primal sketch.

Figure 3.

Formal description of edge detection. Retinal image, I(x, y), is convoluted through the filtering operator ∇²G, where G is a Gaussian and ∇² is a second-derivative (Laplacian) operator. Early vision processes include several filters with different Gaussian distributions, and each produces a different set of zero crossings. Intensity values are interpreted as representing light intensities in the visual field, and the colocated zero crossings are interpreted as representing edges, such as object boundaries.

Figure 4.

Different sets of zero crossings. Image (a) is convoluted with different-sized filters. The zero crossings obtained are shown in b, c, and d. Many of the fine details obtained through the smaller-sized filter (b) are not obtained by the larger-sized filter (c), but some of the zero crossings obtained in c do not appear in the b. From Marr and Hildreth (Reference Marr and Hildreth1980), 201, fig. 6. Reprinted with permission from Royal Society Publishing.

One might think that the job of computational theories stops here, namely, in the formal, function-theoretic specification of the input-output relations. Frances Egan, for one, associates Marr's “computational theory” with “the specification of the function computed” (Reference Egan1991, 196–97). She argues that computational theories provide no more than mathematical specifications: “The top level should be understood to provide a function-theoretic characterization,” and “the theory of computation is a mathematical characterization of the function(s) computed” (Reference Egan1995, 185).Footnote ⁴ As Egan notes, Marr himself refers, in the epilogue of Vision, to the mathematical formula ∇²G*I as the computational description of what the retina does: “Take the retina. I have argued that from a computational point of view, it signals ∇²G*I (the X channels) and its time derivative ∂/∂t(∇²G*I) (the Y channels). From a computational point of view, this is a precise specification of what the retina does” (Reference Marr1982, 337).

I think that Egan captures very well the way Marr characterizes the What element in computational theories. The job of this element is to provide a precise specification of what the system does, and the precise specification of what the retina does is provided by the formula ∇²G*I. However, Egan downplays the fact that there is another component to the computational level, namely, the Why element. Marr refers in the passage to what the retina does but not to why. As we shall see in the next section, Marr stresses that a computational-level theory goes above and beyond the characterization of the what, namely, of the mathematical function that is being computed.

3 Beyond the What: Asking Why

Marr repeatedly associates the computational level with two components, the what and the why. In the introductory chapter of Vision, where Marr presents his “philosophical outlook,” he states that “the most abstract is the level of what the device does and why” (Reference Marr1982, 22). The role of the What element is to specify “what is computed,” whereas the role of the Why element is to demonstrate the appropriateness and adequacy of what is being computed to the information-processing task (24–25). In “Artificial Intelligence: A Personal View,” Marr says that at the computational level, “the underlying nature of a particular computation is characterized, and its basis in the physical world is understood. One can think of this part as an abstract formulation of what is being computed and why” (Reference Marr1977, 37).

When discussing the cash register example (Reference Marr1982, 22–24), Marr says that what is being computed by the device is addition. But he then goes on to state that this characterization is only one-half of the computational explanation: “The other half of this level of explanation has to do with the question of why the cash register performs addition and not, for instance, multiplication when combining the prices of the purchased items to arrive at a final bill” (22). I return to analyze this example below, but it is important to see at this point that Marr refers to the computational level as a level of explanation and that the Why element is an essential part of this computational explanation.

Marr's statements about the why are not merely declarative. The quest for why occurs in his computational theories of vision. Take his theory of edge detection, for example. After characterizing what as being computed in early vision—namely, the colocated zero crossings of ∇²G*I—Marr emphasizes that we still have to confront another critical question: “Up to now I have studiously avoided using the word edge, preferring instead to discuss the detection of intensity changes and their representation by using oriented zero-crossing segments. The reason is that the term edge has a partly physical meaning—it makes us think of a real physical boundary, for example—and all we have discussed so far are the zero values of a set of roughly band-pass second-derivative filters. We have no right to call these edges, or, if we do have a right, then we must say so and why” (Reference Marr1982, 68).

It is not easy to tell what Marr means by this statement. But it is clear enough that he is concerned with the veridical relation between the visual system and the visual scene, for example, between colocated zero crossings that are the basis of edge segments and physical edges such as object boundaries. This concern is emphasized, perhaps more explicitly, by Marr and Hildreth, who say that “the concept of an ‘edge’ has a partly visual and partly physical meaning. One of our main purposes … is to make explicit this dual dependence” (Reference Marr and Hildreth1980, 211).

The specific questions that Marr raises in this respect are (a) why are the zero crossings that result from different-sized filters related to the same feature in the physical environment (see fig. 4)? After all, as Marr puts it, “there is no [a] priori reason why the zero-crossings obtained from the different-sized filters should be related” (Reference Marr1982, 68). And (b) why is this feature often an edge in the physical sense, for example, an object boundary? After all, even if these zero crossings are related to a single feature, there is no a priori reason as to why this feature is a physical edge.

Another, and even more striking, example of the quest for why appears in the theory of stereo vision. As we occasionally experience, there is an angular discrepancy in the position of an object in the two retinal images. This discrepancy is termed disparity. The disparity is usually larger when the object is closer to the eyes (a finger touching your nose) and smaller when it is farther away. The visual system deploys disparity to compute several features, of which the most significant is depth. The first step of this process is matching up elements from the visual scene—that is, finding the two elements, one from the left retinal image and the other from the right retinal image—that correspond to the same object. The difficulty of the task stems, among other things, from the ambiguity of elements in the images and the multiple possibilities of matching the elements.Footnote ⁵

An example is illustrated in figure 5. The four projections (“dots”) in the left eye's view (L₁, … , L₄) can be paired in various ways with the four projections (R₁, … , R₄) in the right eye's view. Of all the pairing possibilities, only the four filled circles are correct, whereas the other 12 open circles are false targets. Still, the visual system solves the correspondence problem, even in highly ambiguous scenes. What explains this remarkable ability?

Figure 5.

Ambiguity in correspondence. Four projections on the left eye's image (L₁, L₂, L₃, L₄) could match any of the four projections (R₁, R₂, R₃, R₄) on the right eye's image. Of the 16 possible pairings, only four are correct (filled circles), while the remaining 12 (open circles) are false targets. Dashed lines signify the amount of (horizontal) disparity; circles (pairs) that are on the same line have the same disparity. From Marr and Poggio (Reference Marr and Poggio1976), 285, fig. 1. Reprinted with permission from American Association for the Advancement of Science.

Marr and Poggio (Reference Marr and Poggio1976, Reference Marr and Poggio1979) argue that the pairing function that provides matching is the one that satisfies two constraints: (a) Uniqueness, a black dot from one image can match no more than one black dot from the other image (this constraint rules out, e.g., the function that matches L₁ to R₁ and also L₁ to R₂), and (b) Continuity, disparity varies smoothly almost everywhere. This constraint rules out functions that match up pairs with very different disparities. Marr (Reference Marr1982, 111–16) demonstrates that these constraints are efficient; they provide necessary and sufficient conditions for matching in most natural scenes.Footnote ⁶

So far, we have dealt with the What element of the computational-level theory. This element states very precisely what the visual system computes, namely, a function that satisfies Uniqueness and Continuity (UC) pairing. This characterization, however, does not exhaust the role of the computational level. Marr argues that a computational level also has to explain why computing this function is appropriate for the information-processing task. The computational level has to explain why the UC-pairing function—and not another pairing function—provides matching. As Marr puts it, “The real question to ask is Why might something like that work? For the plain fact is that if we look just at the pair of images (in fig. 5), there is no reason whatever why L₁ should not match R₃; L₂ match R₁, and even L₃ match R₁” (Reference Marr1982, 112).

Again, it is not immediately apparent what Marr means by this Why question. But I think it is safe to say at this point that Marr thinks that the business of a computational-level theory is not exhausted by the characterization of the mathematical mapping function. Computational theories should do more than that. They should also explain the basis of this mathematical function in the physical world. In what follows, I attempt to explicate what Marr means by a computational-level theory. Specifically, I focus on the following issues: (a) What exactly is the problem that the Why element aims to address (sec. 4)? (b) How does the Why element address this problem (sec. 5)? And (c) how does the Why element combine with the What element to constitute a computational-level theory (sec. 6)?

4 Why Ask Why: The Appropriateness Problem

The picture emerging so far is that the role of computational theories is to analyze information-processing tasks such as edge detection. The role of the What element is to characterize, in mathematical terms, what is computed. In the case of edge detection, the characterization is in terms of the zero crossings of different-sized Laplacian filters of the image. The role of the Why element is to demonstrate the basis of this mapping function in the physical world (Marr Reference Marr1977, 37) or the appropriateness and the adequacy of this mapping to the information-processing task (Marr Reference Marr1982, 24). In the case of edge detection, it is to demonstrate why detecting zero crossings is appropriate for detecting edges. But what does this appropriateness come down to?

I want to suggest that appropriateness comes down to explaining why computing zero crossings is related to edge detection. After all, the mapping process that starts from the retinal image, I, and is described by the colocated zero crossings of ∇²G*I is an “internal” mechanical process that takes place in our brain, whereas the information-processing task is defined, at least partly, by “external” features in the visual field, such as light reflectance, illumination, and object boundaries. We thus should wonder why on earth computing these zero crossings ends up with representations of physical edges: Why does it not end up with a representation of color? Why does it end up with meaningful representations at all?

A similar question arises in the context of stereo vision. The question is why computing the UC-pairing function has anything to do with matching: Why does the process that pairs elements from the two images, and is governed by the Uniqueness and Continuity constraints, end up with matching, namely, a representation of the same physical feature? Why is it that this UC pairing, and none of the other pairing functions, is appropriate for the task of stereo vision, leading to matching?

The problem, in its most general form, is that of explaining why a process that starts from a representation of W₁ (i.e., B₁) ends up with a representation of W₂ (i.e., B₂) and not with a representation of something else. How is it that the B₁–B₂ relations track or mirror the external W₁–W₂ relations? After all, we start the process with B₁, which represents W₁, and then proceed from B₁ to B₂ through a mechanical (causal) process that takes place in our brain. We thus should wonder what is it about the causal relation between B₁ and B₂ that guarantees that the neural representation of W₁ will lead to a neural representation of W₂.

The What element specifies the mathematical relation (function), f, between B₁ and B₂; in the case of edge detection, it is the colocated zero crossings of ∇²G*I. But this does not solve the appropriateness problem; it simply moves the problem one level up. For now, we ask why the f-relation between B₁ and B₂ is appropriate for the information-processing task that is defined in terms of W₁ and W₂: Why does the process that starts from a representation of W₁ (i.e., B₁) and computes a mathematical function f end up with a representation of W₂? Why does the neural process that starts with a representation of light intensities, that is, the retinal image I, and computes the zero crossings of ∇²G*I end up with a representation of physical edges such as object boundaries and not, say, with a representation of the object's colors? Why is it that the f-relations between B₁ and B₂ mirror, as it were, the W₁–W₂ relations?Footnote ⁷

5 Explaining Why: The Role of Physical Constraints

Why does detecting zero crossings help in finding the edges of objects? Why are the zero crossings obtained from different-sized filters related to the same physical feature, and why is it that this feature is (often) a physical edge, for example, the boundary of an object? To answer these questions, Marr famously appeals to physical features in the environment known as natural or physical constraints: “The business of isolating constraints that are both powerful enough to allow a process to be defined and generally true of the world is a central theme of our inquiry” (Reference Marr1982, 23).

In the case of edge detection, Marr (Reference Marr1982, 68ff.) refers to the constraint of spatial localization, which means (in this context) that the things in the world that give rise to intensity changes are spatially localized. Generally speaking, intensity changes should be present under different-sized filters (this will not be the case if two or more local intensity changes are averaged together or two different physical phenomena operate on the same region). Another pertinent physical fact is that intensity changes in the image result from “surface discontinuities or from reflectance or illumination boundaries” (Marr and Hildreth Reference Marr and Hildreth1980, 187). Taken together, it follows that a set of independent zero-crossing segments that have the same position and orientation indicates “the presence of an intensity change in the image that is due to a single physical phenomenon (a change in reflectance, illumination, depth, or surface orientation)” (Marr Reference Marr1982, 70). Thus, the explanation goes, colocated zero crossings usually refer to a single physical feature because the (different) zero crossings point to the same location in the visual field, and in this location we (often) find only one feature that gives rise to intensity changes. This feature is (often) a physical boundary because sharp intensity changes occur along surface discontinuities, and they do not occur along the solid faces of surfaces.

What can we say about this sort of explanation? First, the explanation relates the zero crossings to contingent facts about the actual world, which comes in the form of physical constraints, that is, spatial location. It indicates that we happen to live in a world in which sudden changes in the retinal image (which the zero values of ∇²G*I measure) strongly correlate with object boundaries. We could have lived in a world that consists of surfaces that sharply change reflectance across their solid faces. In such a world, the zero crossings of ∇²G*I would be a poor way of detecting boundaries of objects. Zero crossings would correlate, if anything, with something else. In this sense, the explanation is ecological (Sterelny Reference Sterelny1990).

Second, the explanation does not appeal to adaptive, that is, learning or evolutionary, processes. Marr obviously assumes that the correlation between zero crossings and object boundaries was established by some adaptive-based process. Still, he apparently does not think that the reference to learning and evolution explains why some things can correlate and some not, just as learning and evolution do not explain why the simple perceptron can implement an AND function but not an XOR (exclusive-or) function.

Third, a careful examination reveals that Marr's explanation appeals to similarity between the internal mapping relations and external relations between the features that are being represented. The similarity is not at the level of physical properties. After all, the physiological properties of the brain are quite different from the physical and optical properties that make up our visual field. The similarity is at a more abstract level of mathematical properties. Put differently, the description in terms of zero crossings of ∇²G*I refers to relations at two different levels. It describes internal mapping relations between the retinal and V1 cells. But it also describes relations between light intensities in the visual field (signified by I) and sudden changes in light intensity; these changes happen to occur, at least in our world, along object boundaries. The mathematical relation between the features that are being represented—for example, light reflectance in the visual field and light reflectance along object boundaries—is also that of (extreme points of) derivation.

Thus, Marr not only demonstrates that the internal mathematical function correlates with the contingent world that we live in. He also underscores the basis of this correlation, which is a similarity of mathematical structures. This mathematically based similarity, I maintain, is the key in addressing the appropriateness problem and, hence, in computational-level explanations. In the remainder of this section, I show that this idea of mathematically based similarity is also central in other computational-level theories. In the next section, I spell out the structure of the explanation.

Let us return to the cash register example. The What element characterizes the mapping function from input digits to output digits as addition. We arrive at this characterization when noticing that the machine maps digits to digits and that this mapping satisfies the rules of commutativity, associativity, zero, and inverses. We then ask why computing addition is appropriate for the information-processing task, which is combining the prices of purchased items to arrive at the final bill. Or, as Marr puts it, we ask “why the cash register performs addition and not, for instance, multiplication when combining the prices of the purchased items to arrive at the final bill” (Reference Marr1982, 22). After all, there is no a priori reason for the final bill to be the sum of the purchased items and not their product. We can certainly think of stores in which the cashier executes multiplication and not addition.

In answering this question, Marr demonstrates that the external relation between the final bill and the purchased items in this particular case is also that of addition. Spending enough time around the store, we see that the relation between the purchased items and the final bill is that of addition and not of multiplication. The reason, according to Marr, is that the rules (“constraints”) of purchasing in this store define addition. These are the rules of zero (“if you buy nothing, it should cost you nothing; and buying nothing and something should cost you the same as buying just the something”; Reference Marr1982, 22), commutativity (“the order in which goods are presented to the cashier should not affect the total”), associativity (“arranging the goods in two piles and paying for each pile separately should not affect the total amount you pay”), and inverses (“if you buy an item and then return it for a refund, your total expenditure should be zero”; 23).

Establishing that the relation between the purchased items and the final bill is that of addition, Marr draws the conclusion that the input-output addition mapping in the cash register is appropriate for the task in this particular store. This explanation appeals to the fact that this internal mapping (of addition), defined over digits, corresponds to an external relation between the represented items (in the abstract), namely, between the prices of purchased items and the final bill.

Let us finally turn to stereo vision. We recall that Marr argues that the visual system computes a UC-pairing function. But we also recall that he asks why computing this function is appropriate for matching: why is it that the UC pairing, and not any of the other pairings, provides a solution for the correspondence problem? In answering this question, Marr motivates Uniqueness and Continuity by appealing to physical constraints. Uniqueness (“a black dot from one image can match no more than one black dot from the other image”) is motivated by the spatial localization constraint, which says that “a given point on a physical surface has a unique position in space at any one time” (Marr and Poggio Reference Marr and Poggio1976, 284; see also Marr Reference Marr1982, 112–13). Continuity (“disparity varies smoothly almost everywhere”) is motivated by cohesiveness of matter, which says that “matter is cohesive, it is separated into objects, and the surfaces of objects are generally smooth compared with their distance from the viewer” (Marr and Poggio Reference Marr and Poggio1976, 284; see also Marr Reference Marr1982, 112–13).

We can note that here, too, the reference to physical constraints shows that the internal UC pairing corresponds to an external UC pairing. The similarity is not easily seen, but it is there: what is being represented by each image is an array of light intensities in the visual scene. Recall that these light intensities partially depend on the viewpoint and that the viewpoints of the two eyes are slightly different. What the physical constraints show is that when you apply a UC pairing to these external arrays of light intensities (not to the images), you get matching: the output of the function consists of pairs of elements from the visual scene, whereas the physical constraints dictate that in our world the elements of each pair are really one physical feature.

Marr is unique in making the idea of mathematically based similarity a key feature of computational-level theories. The idea itself, however, is not unique to Marr's theories of vision. It is found in Classical (see discussion of s-representation in Ramsey [Reference Ramsey2007], 77–92) and Connectionist (see discussion in Churchland Reference Churchland2007) models of cognition. It is also found in many computational models in neuroscience, for example, the Zipser-Andersen model (Zipser and Andersen Reference Zipser and Andersen1988; see discussion in Grush Reference Grush, Machamer, Grush and McLaughlin2001). Another example is the oculomotor integrator, whose task is to hold the eyes still between the saccadic movements (Robinson Reference Robinson1989; Seung Reference Seung1996, Reference Seung1998; see discussion in Shagrir Reference Shagrir2010). In this example, both the internal and the external relations are described in terms of integration. The internal mapping relation, from one stable state of the neural memory to another, is described as integration over the pulse saccadic inputs. The external relation between the eye positions (represented by these stable states) is described as integration over the eye velocity with respect to time (which is represented by the pulse inputs).

6 The Structure of Computational-Level Explanations

According to Marr, the job of computational-level theories is to analyze vision as an information-processing task “in a way that is independent of the particular mechanisms and structures that implement them in our heads” (Reference Marr1982, 19). It is time to clarify how the What element and the Why element join forces in providing such an analysis. The gist of the account is this: an information-processing task is mapping from one representation to another (fig. 1), such as in edge detection (fig. 2). The goal of a computational analysis is to explain why the process that starts from a representation of W₁ (i.e., B₁) ends up with a representation of W₂ (i.e., B₂) and not with a representation of something else. A computational-level theory answers this question by pointing to a similarity at the more abstract level between the internal B₁–B₂ relations and the external W₁–W₂ relations. It states that the mathematical B₁–B₂ relations are “isomorphic” (in a sense specified below) to the mathematical W₁–W₂ relations. The role of the What element is to specify the mathematical function that is being computed, namely, the f-relation between B₁ and B₂. The role of the Why element is to demonstrate that the W₁–W₂ relation is also an f-relation.

Putting it schematically, the structure of a computational-level explanation is, ideally, as follows:

Therefore, i(B₂) = W₂.

The first premise states that B₁ represents W₁. In our example, it states that the intensity values, I(x, y), represent light intensities in the visual scene, which result from such features as light reflectance, illumination, geometry, and viewpoint. The second premise states that the mathematical relation between B₁ and B₂ is f. Characterizing this function is the business of the What element. The What element provides a description of the internal mapping relations between the abstract properties of B₁ and B₂. In our example, the What element specifies the mapping relations between abstract properties of the photoreceptors, ganglion cells, and cells in V1 (abstracted from the electric activity of these cells) in terms of zero crossings of ∇²G*I.

The job of the Why element is to establish the third premise, namely, to show that the W₁–W₂ relations are also f-relations. Establishing this premise is often trivial, but it sometime takes considerable sophistication and effort. Edge detection is an example. As Marr says, it does not immediately follow that if the term I(x, y) refers to the array of light intensities in the visual field, then the colocated zero crossings of the different-scale filtering of ∇²G*I stand for physical edges. That they do is a contingent fact about our visual environment and should be argued for. The argument, we saw, appeals to the physical constraints, which are facts and assumptions about our physical world.

Explicating the structure of explanation reveals that the first three premises do not suffice to derive the conclusion. There is another, implicit, assumption here, which is the fourth premise. This premise asserts that the visual system is a representational system in some very strong sense: it respects, as it were, mathematical relations, or at least some mathematical relations (i.e., f). Put differently, the premise states that the representation function, i, is an isomorphism over the (mathematical) relation f. Marr never makes this assumption explicit, nor does he argue for it.Footnote ⁸ But the assumption is essential for the derivation of the conclusion.Footnote ⁹

This sketch of computational explanations requires further elaboration and explication. Here, I will make two brief comments. One is about abstraction: the function f is mathematical in the sense that its domain and range are mathematical entities such as real numbers, geometrical relations, set-theoretic structures, and so forth. Thus, the function f relates numbers (or other mathematical entities) and not physical entities. These entities are mathematical values, as magnitudes that abstract from pertinent physical properties. At one level, the function relates numbers that abstract from the representations (e.g., electrical cellular activity). At another, it relates magnitudes that abstract representational contents (e.g., distal objects in the environment). The other comment is about approximation: a visual system, as a biological system, seldom computes the function f that is stated by the computational-level theory; it only approximates it. Likewise, the function f is, at best, an approximation of the distal relation. A computational-level theory thus relates the ideally computed function f, with an (ideal) external f-relation.Footnote ¹⁰

7 Computation and Intentionality

In closing, I compare my interpretation of Marr with other interpretations. My aim, however, is neither to provide an exhaustive survey of all other interpretations nor to criticize them. It is to locate my interpretation in the wider philosophical landscape and to clarify the ways in which Marr's computational-level theories can be seen as intentional.

Interpretations of Marr have focused on issues of explanation, individuation, and methodology. Let us start with explanation. It is widely agreed that, according to Marr, a theory of vision should analyze a visual, information-processing task. The debates concern the role of computational-level theories in this analysis. Most philosophers take it that the role of all computational-level theories is to provide an information-processing description of the task, in terms of the visual (representational) content of neural states. In other words, the claim is that the main job of computational-level theories is “the specification of the explanandum—the cognitive task that we are attempting to explain. Marr calls this the ‘computational’ level, where the specification is typically an input-output function” (Ramsey Reference Ramsey2007, 41). Bermúdez also writes that “a computational analysis will identify the information with which the cognitive system has to begin (the input to that system) and the information with which it needs to end up (the output from that system)” (Reference Bermúdez2005, 18). And Horst (Reference Horst and Zalta2009) concludes that “at the highest level was a specification of what task a system was designed to perform: for example, in the case of vision, to construct a three-dimensional representation of distal stimuli on the basis of inputs to the retina.”

It should be noted that interpreters of Marr who adopt this line of thinking believe that Marr confuses the “computational” with the “representational.” Sterelny, for example, says that “Marr, very confusingly, calls it the ‘computational’ level” (Reference Sterelny1990, 46). Dennett, after associating Marr's computational level with his intentional level, says that “this specification was at what he [Marr] called, misleadingly, the computational level” (Reference Dennett, Prawitz, Skyrms and Westerståhl1994, 681). Ramsey notes that “this [computational] label is somewhat misleading” (Reference Ramsey2007, 41 n. 3), and Horst (Reference Horst and Zalta2009) says that “this level Marr (somewhat unfortunately) called the ‘computational level.’” But, of course, Marr is confused only if he takes his computational-level theories to provide such information-processing descriptions of the task.

My view is that computational-level theories do not aim to provide such intentional descriptions. The characterization of the task in terms of informational content is often made before we invoke the computational-level theory; in the case of edge detection it is being made at the level of neurophysiology. That photoreceptors are sensitive to light reflectance, that information from the retina arrives to V1, and that cells in V1 are sensitive to oriented lines were all discovered by neurophysiologists, using techniques such as single-cell recording, long before Marr invoked his computational theories.Footnote ¹¹ Computational theories, according to Marr, aim to explain something about these information-processing tasks. They aim to explain why the neural process that starts with the representation of light intensities ends up with a representation of physical edges.

Other interpreters also think that the computational level plays an explanatory role in a theory of vision. Shapiro (Reference Shapiro1997), for example, writes that “at the computational level of theory the theorist describes what I shall call chief tasks and service tasks. … The chief task of the visual system is the derivation of 3-D shape representations from information encoded in 2-D images. Service tasks are those tasks the completion of which contribute to the achievement of the chief task” (134). In particular, he argues, the information-processing description of the service tasks, in terms of informational content, contributes to the understanding of the chief task. This is in accord with the functional picture of computational explanations, according to which the capacity of a system (chief task) is explained in terms of the capacities of the components (service tasks) of which it is composed.Footnote ¹²

I agree with Shapiro that the information-processing descriptions of service tasks (e.g., stereo disparity) account for the chief task (e.g., stereo vision). But I insist that the paradigm cases of computational theories that Marr and his students advance—edge detection, stereo disparity, and structure from motion—are all (service) tasks that do not comfortably break down to further service tasks. The explanation of these tasks is very different and refers to two kinds of formal relations. One is the inner input-output mathematical relation, and the other is the outer mathematical relation between what is being represented by the inputs and the outputs. The first sort of relation, of the input-output function, is not intentional. The latter sort can be seen as some kind of mathematical or formal content, in that these mathematical relations are abstracted from representational contents, for example, physical edges.Footnote ¹³ Thus, if computational explanations are intentional, it is by virtue of referring to these features—that is, the formal content—and not to specific content.

Egan presents a very different picture according to which the aim of computational-level theories is to provide a mathematical characterization of the (input-output) computed function. The intentional content is invoked to explain how the computed mathematical function can serve a cognitive function in a particular context. Thus, Egan argues that “an explanation of how the visual system detects the depth of the scene from information contained in two-dimensional images is forthcoming only when the states characterized in formal terms by the theory are construed as representations of distal properties” (Reference Egan1995, 190).Footnote ¹⁴

I agree that one aim of computational-level theories is to characterize the (mathematical) computed function; this is the job of the What element. I also agree that computational-level theories are concerned with the embedment of the (computing) system in its environment; indeed, the role of the Why element is to explain the physical basis of the (computed) mathematical function in the physical world. Still, I resist the idea that the computational explanations answer this embedment question by specifying the intentional content of visual states. The embedment question is not “What is represented by the zero crossings?” For one thing, we know the answer to this before we invoke the computational-level theory, from the electrophysiological experiments; for another, Marr had no reason to make his dramatic statements about the why, if the role of the Why element is just to specify the content of the representations. The issue, rather, is explaining why the outputs of the process represent what they represent: why computing zero crossings leads to representations of edges and not to something else. And this explanation is provided not by referring to the intentional content of the visual states but by revealing the structural similarities between the internal input-output relations and the pertinent external relations.Footnote ¹⁵

Much of the philosophical debate about Marr concerns individuation. Most of Marr's interpreters argue that his computational theories are intentional, in the sense that they make an essential reference to specific visual content. The claim, in other words, is that content affects the computational (type) identity of visual states.Footnote ¹⁶ Egan, in contrast, argues that computational types are affected by mathematical content but not by specific visual content: “For the purpose of individuation, the precise mathematical description given by the theory of computation is the description that counts” (Reference Egan1995, 186). She thus concludes that computational theories are individualistic (see also Egan Reference Egan2010).

I agree with Egan that specific content does not always make a computational difference, namely, that two systems can be computationally identical even if they carry different content. I also agree with Egan that computational (“formal”) types are affected by mathematical content in the sense that they refer to mathematical relations that are abstracted from representational content, for example, physical edges. Still, it does not follow that specific content does not affect computational individuation (and, hence, that computational theories cannot be externalistic). When you abstract from different contents, you sometimes get the same mathematical content, in which case the systems are computationally identical, but you sometimes get different mathematical content, in which case they are not.Footnote ¹⁷

Finally, I discuss methodology. Marr famously advocates a top-down methodology, wherein the top level is the computational level. Most interpreters of Marr hold that the top level specifies an intentional description of the task; they think that the methodology is top down in the sense that the intentional description restricts the algorithm (see Shapiro Reference Shapiro1997, 137). I agree that the top computational level provides guidance to the specification algorithms, although I think that the specification is function theoretic (mathematical) and not information processing (intentional). The main difficulty with the suggested interpretation, however, lies elsewhere: it is that the intentional (information-processing) description itself is very often provided through electrophysiological single-cell-recording experiments that combine “implementation,” that is, action potentials, with environmental stimuli. This makes the methodology no more top down than bottom up.Footnote ¹⁸

According to my interpretation, Marr's contentious claim about methodology refers to the specification of the mathematical function that is being computed. The obvious method to arrive at this mathematical function is through abstraction from neurological properties (bottom up). But Marr argues that this method is not effective and that the more effective way is to start with meaningful (physical) constraints on the relations between the features that are being represented.Footnote ¹⁹ In the case of edge detection, we start with the fact that there are sudden changes in light reflectance along physical edges, and we infer from this that the pertinent mathematical relation is that of a derivative. The next step is to look for this mathematical relation inside the brain. In the case of edge detection, we look for derivation in early visual processes.

When properly understood, we can see that Marr's methodological framework is still widespread in computational neuroscience. It is found, for example, in the study of the oculomotor integrator (mentioned above). The first stage in the investigation was the discovery, through single-cell experiments, that the external relation between the features represented by inputs (i.e., transient eye velocity) and outputs (i.e., eye position) is that of integration (in fact, this is the reason the system was called integrator in the first place).Footnote ²⁰ Realizing this, theoreticians moved to the next step of looking for an integration relation in the cellular activity of the representing cells (see Robinson Reference Robinson1989; Seung Reference Seung1996, Reference Seung1998). Such studies not only materialize Marr's conceptual and methodological maxims. They also combine experimental studies, alongside sophisticated theoretical work, which are exemplars of Marr's vision.