Introduction
The dissociation of a figure from background is an important part of object perception. Figure processing is necessary for us to detect, recognize, and interact with objects and entities in our environment. Not surprisingly then, humans are extremely sensitive to figure information, detecting and discriminating shapes across multiple cue types such as regional contrast of luminance, texture, color, motion, and binocular disparity (Regan, Reference Regan2000) (see Fig. 1). We are so sensitive to potential figures that even cue-impoverished stimuli such as Kanizsa’s (Reference Kanizsa1979) triangle nonetheless yield strong percepts of modally completed “illusory” figural contours (see Fig. 2A). Attempts to understand how the brain instantiates our perception of figures have collectively provided some important cortical regions of interest and their associated roles in figure processing.
Examples of figure contrast. Most experiments on figure contrast investigate sensitivity to only one type of contrast, such as (A) luminance or (B) texture, yet real figures are defined by multiple combined cues. (C) This schematic example shows a figure defined by texture, luminance, and spatial frequency contrast, whereas (D) shows a more realistic figure defined by luminance and texture contrast, plus some depth and shading cues.
Gestalt illusory shapes. In (A), the arrangement of the pacman inducers elicits the perception of an illusory white triangle occluding three black circles. The triangle results from modal completion of the gapped boundaries into a coherent figure contour, while the perception of the pacmen as occluded circles represents amodal completion of their hidden surfaces. In (B), the white triangle shape is completed amodally, and appears as if viewed from three holes in a wall.
At the heart of the field of neuroscience is an attempt to understand the relationship between an animal’s nervous system and its behavior. Similarly, experiments in psychophysics seek to investigate the manner in which measures of perceptual discrimination reflect constraints of underlying neural mechanisms. In this spirit, Teller (Reference Teller1984) outlined five families of “linking propositions” that can be used to infer neurological substrates of psychophysical data. Although some researchers combine psychophysical and neurophysiological recording methods to explore these links directly, many experiments adhere to only one methodology. In this case, linking “hypotheses” may be formed by bridging separate studies to suggest possible behavior/brain relationships. With regard to figure processing in humans, some studies do combine a direct behavioral measure with neuroimaging data, whereas others use passive viewing methods. To date, the most common task is a simple figure detection task. This does not allow for a specific investigation of individual processes underlying figure perception, but does ensure participants’ perception of relevant stimuli relative to distractors, as well as attentional engagement. While a similarly explicit link to behavior cannot be made with data from the passive viewing studies, in some cases, such comparisons of data from task-based versus passive viewing conditions have not differed (Murray et al., Reference Murray, Wylie, Higgins, Javitt, Schroeder and Foxe2002; Bakar et al., Reference Bakar, Liu, Conci, Elliott and Ioannides2008; Guzzon & Casco, Reference Guzzon and Casco2011; Appelbaum et al., Reference Appelbaum, Ales and Norcia2012), but see an exception in the case of peripheral viewing conditions (Bakar et al., Reference Bakar, Liu, Conci, Elliott and Ioannides2008). More generally, we can consider that passive viewing experiments are nonetheless eligible to support links to related task-based neuroimaging and psychophysical studies by analogy. Studies on nonhuman animals such as monkeys and cats also examine direct links between behavioral and physiological data, and these can also be compared by analogy to the data of human subjects (see Tootell et al., Reference Tootell, Tsao and Vanduffel2003; Orban et al., Reference Orban, Van Essen and Vanduffel2004). Such comparisons ideally allow us to understand how single-unit spiking activity—reflective of neuronal action potentials—relates to the population level activity that underlies both VEP and fMRI signals for noninvasive human studies. Although the differences between the recording methodologies serve to qualify interpretations of findings with regard to figure processing, there are nonetheless encouraging similarities in their conclusions with regard to regions of interest and their functions (e.g., Kriegskorte et al., Reference Kriegeskorte, Mur, Ruff, Kiani, Bodurka, Esteky, Tanaka and Bandettini2008).
The primate visual system is understood to be roughly hierarchical in organization, proceeding from striate cortex, to extrastriate regions, to increasingly higher-tier regions in lateral occipital (LO), temporal, or parietal cortex. This procession is not purely feed-forward, however, and feedback connections to lower visual areas have been shown to be very important for figure processing, as will be discussed. In fact, several proposed neural models of object and texture processing focus on statistical processing of natural image features rather than a hierarchically organized feed-forward circuit (e.g., Simoncelli & Olshausen, Reference Simoncelli and Olshausen2001; Naselaris et al., Reference Naselaris, Prenger, Kendrick, Oliver and Gallant2009). As an organizing principle for the contents of our review, however, we have opted to follow a somewhat conventional hierarchical scheme.
In Fig. 3, we propose a provisional model of distinct cortical processing stages related to figure perception. The stages of the model are not without controversy or disagreement, and these will be addressed accordingly, in turn, in the sections below. The first stage represents the processing of basic visual features at the local level of individual cells in primary visual cortex (V1); the second stage represents an early segmentation of a visual scene for salient regions; the third stage represents the interpolation of a two-dimensional figure surface, including the contours that distinguish the surface from surrounding regions; the fourth stage concerns selectivity for figure shapes, enabling discrimination and generalization; the fifth stage represents a volumetric interpolation of an object’s surface; and the sixth represents a pair of parallel stages for volumetric shape categorization and detecting changes in the depth structure of a scene. Particular focus will be placed on the earlier stages of the model that relate most readily to two-dimensional figures, although the distinction of 2D from 3D image processing is not always clear, as will be made apparent.
Model of figure processing.
Local feature sensitivity
The first stage in our model is the early processing of basic visual features, as occurring in primary visual cortex (V1) (Hubel & Wiesel, Reference Hubel and Wiesel1959, Reference Hubel and Wiesel1968). There is a large body of work that is impressive in its description of a retinotopically organized, multiplexed reduction of the visual image into simple features. At this early stage, local differences of luminance, color, orientation, or motion properties in the visual field excite the cells in V1 with receptive fields that correspond to the retinal area of stimulation. These local discrepancies provide “contrast” cues that are important for the detection of object edges (Livingstone & Hubel, Reference Livingstone and Hubel1988). While the most formative work on V1 is based upon single-unit recordings of cat and monkey cortex, fMRI studies on humans also demonstrate that area V1 is similarly important for early feature processing (e.g., Engel et al., Reference Engel, Glover and Wandell1997; Tootell et al., Reference Tootell, Hadjikhani, Hall, Marrett, Vanduffel, Vaughan and Dale1998; Schluppeck & Engel, Reference Schluppeck and Engel2002; Swisher et al. Reference Swisher, Gatenby, Gore, Wolfe, Moon, Kim and Tong2010). Additionally, lower visual cortex is important for the perception of figure contours. For instance, single-unit recording studies on monkeys have demonstrated for V1 neurons sensitivity to more global features such as a coherent versus incoherent Kanizsa shape (Lamme et al., Reference Lamme, Van Dijk and Spekreijse1993). Importantly though, the sensitivity to such stimuli has been shown to occur in V2 before V1, implying an early influence of feedback processing (Lee & Nguyen, Reference Lee and Nguyen2001). Generally, the contribution of lower cortical regions to our detailed perception of figural contours is thought to depend upon feedback from higher visual areas (Grosof et al., Reference Grosof, Shapley and Hawken1993; Heider et al., Reference Heider, Meskenaite and Peterhans2000; Ramsden et al., Reference Ramsden, Hung and Roe2001; Bauer & Heinze, Reference Bauer and Heinze2002; Roelfsema et al., Reference Roelfsema, Lamme and Spekreijse2004; Li et al., Reference Li, Piëch and Gilbert2006; Roelfsema et al., Reference Roelfsema, Tolboom and Khayat2007; Likova & Tyler, Reference Likova and Tyler2008; Strother et al., Reference Strother, Lavell and Vilis2012). This will be discussed in our section on figure contour interpolation. We argue, however, that there is a processing stage that is intermediate to local feature processing and contour interpolation. This stage is for the rough spatial segmentation of a scene into salient regions.
Scene segmentation of salient regions
After basic feature processing, the next stage of the model relates to the segmentation of a visual scene. In terms of its underlying neural processes, segmentation is largely thought to depend on a feed-forward progression from V1 to extrastriate regions (Appelbaum et al., Reference Appelbaum, Ales and Norcia2012; Poort et al., Reference Poort, Raudies, Wannig, Lamme, Neumann and Roelfsema2012; Romeo et al., Reference Romeo, Arall and Supèr2012), which exhibit progressively larger receptive fields (Tootell et al., Reference Tootell, Switkes, Silverman and Hamilton1988). Neural responses in regions V1–V4 to segmenting stimuli, most frequently figures defined by texture contrast, have been shown in to be unaffected by attentional demands, with differences between attention conditions being observed only among later responses in higher dorsal and ventral regions (Scholte et al., Reference Scholte, Witteveen, Spekreijse and Lamme2006; Martinez et al., Reference Martinez, Ramanathan, Foxe, Javitt and Hillyard2007; Appelbaum & Norcia, Reference Appelbaum and Norcia2009). Thus, scene segmentation has not been linked to conscious awareness but instead may operate early to cue attention and fixation for later conscious processing. Consistently, a behavioral study by Kimchi and Peterson (Reference Kimchi and Peterson2008) found that altering the figure-ground information in the surround during a difficult task in central vision diminished performance, even without awareness of any alteration. Many researchers describe this early preattentive stage as “figure/ground segmentation,” because, in most cases, the region of the scene that is segmented corresponds to a figure. However, we will avoid this description due to its association with continuous figure contours, a topic that will be addressed in the Figure contour interpolation section. Instead, like Stanley and Rubin (Reference Stanley and Rubin2003), we opt to use the term “salient region” as a cue for image segmentation, which may or may not result in subsequent figure contour interpolation.
Early work on texture segmentation focused on the “pop-out” properties of targets in complex displays (e.g., Julesz, Reference Julesz1981; Nothdurft, Reference Nothdurft1991), and this may relate more generally to early visual cueing to a region of a scene rather than specifically to figure surfaces. Similarly, Kanizsa illusory figures have been shown to exhibit pop-out salience in visual search tasks (e.g., Davis & Driver, Reference Davis and Driver1994; Senkowksi et al., Reference Senkowski, Röttger, Grimm, Foxe and Herrmann2005), leading some researchers (Gurnsey et al., Reference Gurnsey, Pearson and Day1996; Stanley & Rubin, Reference Stanley and Rubin2003) to investigate what aspect of the stimuli drives the pop-out effect. These experimenters each manipulated classic Kanizsa stimuli to diminish the perception of illusory contours (see Fig. 4 for an example) and found that pop-out salience was identical between the classic and the altered Kanizsa stimuli. Because the black inducers of the altered stimuli maintained their closed configuration, the stimuli served as “salient regions” for segmentation in a visual search, even though continuous illusory contours were not perceived. Thus, the sensitivity of the segmentation mechanism to the local features in the global configuration is broad, apparently not requiring the highest spatial precision.
A variant of the Kanizsa square with rounded, slightly misaligned inducers. Despite the absence of illusory contours, the salient region of configural closure is preserved. Because of this, it can be segmented from a scene of randomly oriented inducers.
This broad sensitivity to spatial configuration likely reflects a mechanism with low spatial frequency tuning and that is biased toward stimuli outside of central visual field. Psychophysical studies that directly investigated the effect of eccentricity on sensitivity to texture pop-out found that detection was optimal in the periphery (Fiorentini, Reference Fiorentini1989; Meinecke, Reference Meinecke1989). Related to this, an eye-tracking study by Biber and Ilg (Reference Biber and Ilg2008) found that pursuit initiation to illusory contour stimuli occurs before contour completion can take place, suggesting that segmentation facilitates fixation upon a salient region and that the perception of a figure’s contours requires the high spatial precision afforded by central viewing (Hirsch & Curcio, Reference Hirsch and Curcio1989).
The role of different eccentricity biases across the visual cortex perhaps deserves more attention with regard to figure and object processing. Many studies on the neural correlates of segmentation (e.g., Roelfsema et al., Reference Roelfsema, Lamme and Spekreijse1998; Scholte et al., 2005, Reference Scholte, Jolij, Fahrenfort and Lamme2008; Appelbaum & Norcia, Reference Appelbaum and Norcia2009; Appelbaum et al., Reference Appelbaum, Ales and Norcia2012; Poort et al., Reference Poort, Raudies, Wannig, Lamme, Neumann and Roelfsema2012; Romeo et al., Reference Romeo, Arall and Supèr2012) rely on fixed viewing conditions that require the subject to foveate on the center of a figure or on a figure’s contour. (Scholte et al., Reference Scholte, Witteveen, Spekreijse and Lamme2006, Reference Scholte, Jolij, Fahrenfort and Lamme2008; Appelbaum & Norcia, Reference Appelbaum and Norcia2009; Appelbaum et al., Reference Appelbaum, Ales and Norcia2012). Others recorded from cells with receptive fields eccentric to a figure region (Roelfsema et al., Reference Roelfsema, Lamme and Spekreijse1998; Poort et al., Reference Poort, Raudies, Wannig, Lamme, Neumann and Roelfsema2012; Romeo et al., Reference Romeo, Arall and Supèr2012), but the effect of eccentricity was not directly investigated. However, there are some important exceptions. For instance, McMains and Kastner (Reference McMains and Kastner2011) investigated the effect of eccentricity on BOLD responses to Kanizsa shapes and some deformed variants. The authors found that the typical higher-level lateral occipitotemporal cortex responses to centrally placed coherent Kanizsa shapes were not observed when the shapes were presented outside of the central visual field. Eccentric stimuli activated lower-to-middle-tier extrastriate regions (V2–V4) instead.
Similarly, Caplovitz et al. (Reference Caplovitz, Barroso, Hsieh and Tse2008) found that the synchronized onset of four small circle stimuli in the near periphery (about 6° from fixation) activated areas V2 to V4, but not higher ventral regions; while Wong et al. (Reference Wong, Aldcroft, Large, Culham and Vilis2009) employed a similar paradigm with central object viewing and did find significant activity in the LO complex, the group of areas in ventral visual cortex that have been found to respond selectively for coherent versus scrambled object stimuli (Malach et al., Reference Malach, Reppas, Benson, Kwong, Jiang, Kennedy, Ledden, Brady, Rosen and Tootell1995; Grill-Spector et al., Reference Grill-Spector, Kourtzi and Kanwisher2001; Kourtzi & Kanwisher, Reference Kourtzi and Kanwisher2001). Consistently, higher-tier ventral areas such as those in the LO complex have demonstrated a general sensitivity bias for stimulation in the central visual field, a bias that results from the higher spatial resolution of the fovea and is also shaped by visual experience (Levy et al., Reference Levy, Hasson, Avidan, Hendler and Malach2001; Hasson et al., Reference Hasson, Levy, Behrmann, Hendler and Malach2002; Malach et al., Reference Malach, Levy and Hasson2002; Larsson & Heeger, Reference Larsson and Heeger2006). Regions V1 to V3 (and possibly V4) may therefore be important for the detection of relevant stimuli in the peripheral or parafoveal visual field. The resulting scene segmentation may facilitate visual attention and foveal orienting that allows for a more precise interpolation of the figure via higher visual regions such as those in the LO complex.
Aside from the feed-forward progression of inputs from V1 to V4, another region that may play an important role in scene segmentation is the middle temporal area (MT) of macaques, known as V5 or hMT in humans (Born & Bradley, Reference Born and Bradley2005). This crudely retinotopic area is thought to be crucial for the integration of local motion signals into globally coherent patterns (Adelson & Movshon, Reference Adelson and Movshon1982; Newsome & Pare, Reference Newsome and Pare1988; Born & Tootell, Reference Born and Tootell1992), but it has since been shown that MT may only integrate motion across regions of space rather than across the entire visual field (Majaj et al., Reference Majaj, Carandini and Movshon2007). The functional organization of some MT cell populations—namely, the opponent-tuning for center and surround receptive fields—has led investigators to conclude that the area is necessary for segmentation (Xiao et al., Reference Xiao, Raiguel, Marcar and Orban1997; Masson et al., Reference Masson, Mestre and Stone1999; Born et al., Reference Born, Groh, Zhao and Lukasewycz2000; Born & Bradley, Reference Born and Bradley2005; Huang et al., Reference Huang, Albright and Stoner2007). Indeed, MT has demonstrated sensitivity to motion contrast stimuli (Marcar et al., Reference Marcar, Xiao, Raiguel, Maes and Orban1995; Pack et al., Reference Pack, Gartland and Born2004) and binocular disparity, a cue for object depth (Bradley & Andersen, Reference Bradley and Andersen1998; Deangelis & Newsome, Reference Deangelis and Newsome1999; Krug & Parker, Reference Krug and Parker2011). More specifically, however, the connective properties of MT—for example, direct connections to areas important for eye movements, such as the frontal eye fields, superior colliculus, and dorsolateral pons (Born & Bradley, Reference Born and Bradley2005)—make the area a possible candidate for the initiation of ocular pursuit to a salient region of a scene. The precise contribution of MT to segmentation, however, remains to be fully understood, and the region will also be discussed with regard to contour interpolation and changing scenic depth structure.
Thus, after an initial, automatic feed-forward progression of object processing in visual cortex, the segmentation stage may lead to a more complex interaction between higher and lower cortical regions. The first stage of the model that depends on this complex interaction concerns the interpolation of a figure’s surface contours, discussed below.
Figure contour interpolation
After the spatially rough segmentation of a scene, it is necessary to perceive certain regions within a scene as representing the surfaces of objects, including the contours that separate a figure from its background. The process is referred to here as “interpolation” because our perceptual construction of figure surfaces has been shown to occur readily even with limited visual information (e.g., Figs. 1 and 2). Nevertheless, in natural vision, objects can have clearly defined boundaries that separate the objects’ surfaces from their surroundings. In uncluttered scenes, the contours of an object will close into a bounded two-dimensional image on the retina. These boundaries can be obscured, however, by occlusion in depth, which is discussed in the 2D shape sensitivity section. In addition to this information, objects provide a multitude of cues—such as regional contrast of luminance, color, texture, motion, and disparity—that consistently conform to, and thus reinforce, the spatial boundaries of their contours (Regan, Reference Regan2000, Fig. 1).
From these many potential cues, the surface of a figure can be constructed by the visual system. Yet even in images that rely on only one or a few cues, there is a salient impression of continuous figure contours, even when there are local border discontinuities. Figure contour interpolation processes thus lead to the perceptual enhancement of an object’s contours, which is also thought to be necessary for the illusory contours of a Kanizsa triangle (Fig. 2A). On the other hand, it is possible to find stimuli that have clearly defined line boundaries yet have ambiguous or competing “solutions” for different figure/ground relationships. Hence, the perceived ownership of the boundaries can be manipulated, or even shift from one rivalrous solution to another, as with Rubin’s famous face/vase illusion (Fig. 5). We suggest that this reflects the same contour interpolation mechanism that contributes to perception of texture contrast displays and the Kanizsa shapes. In addition, these manipulations of border ownership (BOWN) perception reflect a mechanism that relies upon the global configuration of a scene for the assignment of a region as a figure, and the interpolation of the figure’s surface boundaries, at least in two dimensions.
Rubin’s face/vase illusion. The image above can be seen as either a white vase in the center or two black faces to each side, but never both percepts at once. This is because the vase and the faces are defined by the same line boundaries and because the perception of any region as a figure requires the “ownership” of a boundary as the contours of the figure. Thus, the vase and faces compete for “border ownership,” and the figure/ground organization changes with each perceptual shift in border assignment.
In principle, how might we identify a contour interpolation mechanism? First, it might exhibit identical response sensitivity across different figure cues. Given that figure cues from different types of figure contrast (e.g., Fig. 1) usually co-occur in natural images, a contour interpolation mechanism would likely show similar sensitivity to figure information across cue types, sometimes referred to as “cue invariance” (Albright, Reference Albright1992; Sary et al., Reference Sary, Vogels and Orban1993). Aside from being cue-invariant, a mechanism that triggers contour interpolation might exhibit a response tuning across parametrically varied local contour strength that is nonmetric, but rather behaves in closer analogy with perceptual detection of figures. This would reflect an all-or-none process with a specific minimum value of figure size or contrast magnitude required to achieve interpolation. Naturally, such “categorical” tuning of responses to figure stimuli should be identical across cue types, as is generally the case behaviorally.
While there is some evidence that dorsal regions such as hMT+ may contribute to the processing of figure contours (see Likova & Tyler, Reference Likova and Tyler2008), their roles may be limited to dynamic figure contrast, rather than serving as a general figure contour interpolation mechanism. In comparison, we do have evidence that more ventral areas in lateral visual cortex may be important for general object processing, as well as for figure contour interpolation. As mentioned above, the LO complex is a group of areas in human ventral visual cortex that have been found to respond selectively for coherent versus scrambled object stimuli (Malach et al., Reference Malach, Reppas, Benson, Kwong, Jiang, Kennedy, Ledden, Brady, Rosen and Tootell1995; Grill-Spector et al., Reference Grill-Spector, Kourtzi and Kanwisher2001; Kourtzi & Kanwisher, Reference Kourtzi and Kanwisher2001). The complex includes the lateralmost parts of occipital cortex (called LO), as well as the likely homologue to monkey inferotemporal cortex (IT) and parts of the fusiform gyrus (Tanaka, Reference Tanaka1997; Tootell et al., Reference Tootell, Tsao and Vanduffel2003; Orban et al., Reference Orban, Van Essen and Vanduffel2004; Preston et al., Reference Preston, Li, Kourtzi and Welchman2008). The anteriormost regions of LO complex such as the fusiform gyrus have been implicated in object processing that is more advanced than the extraction of simple figures from a scene. This includes the recognition of objects by a common category (e.g., Kriegskorte et al., Reference Kriegeskorte, Mur, Ruff, Kiani, Bodurka, Esteky, Tanaka and Bandettini2008) and viewpoint-independent object shape sensitivity (e.g., James et al., Reference James, Humphrey, Gati, Menon and Goodale2002; Vanrie et al., Reference Vanrie, Béatse, Wagemans, Sunaert and Van Hecke2002; Kim & Biederman, Reference Kim and Biederman2012), which will be discussed respectively in the 2D shape sensitivity and Volumetric surface interpolation sections.
In addition, there is evidence that the LO complex may also include a contour interpolation mechanism. In fMRI studies in monkeys (Sereno et al., Reference Sereno, Trinath, Augath and Logothetis2002) and humans (Grill-Spector et al., Reference Grill-Spector, Kushnir, Edelman, Itzchak and Malach1998, Reference Grill-Spector, Kushnir, Edelman, Avidan, Itzchak and Malach1999; Vinberg & Grill-Spector, Reference Vinberg and Grill-Spector2008), for instance, the LO complex has demonstrated similar responses across different figure contrast cues, as well as for line-defined and illusory Kanizsa shapes, thus suggesting a cue-invariant sensitivity to figures. With regard to response tuning across parametrically varied object stimulus parameters, Mendola et al. (Reference Mendola, Dale, Fischl, Liu and Tootell1999) and Altschuler et al. (Reference Altschuler, Molholm, Russo, Snyder, Brandwein, Blanco and Foxe2012) varied the size of Kanizsa figure stimuli, and both sets of authors observed response functions in the LO complex that were size-invariant. The responses were interpreted to reflect a categorical sensitivity to object information. Similarly, Fesi et al. (Reference Fesi, Yannes, Brinckmann, Ales, Norcia and Gilmore2011) compared VEP responses across types of motion contrast, as well as across contrast magnitudes, and found that responses in LO channels exhibited a categorical tuning function that likely reflected a threshold of figure contrast necessary to elicit a response, while dorsomedial channels showed a metric sensitivity that was similar across contrast cue types. In comparison, Thielscher et al. (Reference Thielscher, Kölle, Neumann, Spitzer and Grön2008) varied textural contrast, and argued that their results demonstrated a metric sensitivity to figure information in V4 and LO-complex. However, although the BOLD responses in these regions were parametrically ordered according to magnitude, we contend that the response functions reported by the authors are nonetheless quite compatible with an interpretation of categorical sensitivity rather than metric. Future research will be necessary to settle the matter, but it seems as if V4 and regions of the LO complex demonstrate sensitivity functions that support their consideration as possible mechanisms for contour interpolation.
In another line of evidence, the LO complex has shown responses to contour stimuli that are sustained and modulated by both spatial and object attention (Kourtzi & Huberle, Reference Kourtzi and Huberle2005; Martinez et al., Reference Martinez, Ramanathan, Foxe, Javitt and Hillyard2007), supporting its role as a top-down influence on lower visual areas for contour cues, as opposed to the more automatic nature of a scene segmentation mechanism (Appelbaum & Norcia, Reference Appelbaum and Norcia2009; Keane et al., Reference Keane, Mettler, Tsoi and Kellman2011). Interestingly, Stanley and Rubin (Reference Stanley and Rubin2003) argued that the LO complex is not important for contour interpolation but may instead be involved in scene segmentation. The authors found that BOLD responses in the LO complex were similar for classic Kanizsa shapes and altered variations with preserved salient regions (Fig. 4). The authors’ conclusion, however, was based on separate analyses of scans for the classic and altered conditions, and so the pattern of BOLD activation for a direct comparison of the conditions remains to be formally addressed. Also, in terms of the temporal dynamics of the responses, Shpaner et al. (Reference Shpaner, Murray and Foxe2009) conducted a similarly designed VEP experiment, and found that early evoked responses among LO channels were stronger for Kanizsa shapes than for the modified variants, while dorsomedial occipital responses were similar across the stimulus types, as were later LO channel responses. It remains likely then that regions in the LO complex are responsible for contour interpolation, while dorsal extrastriate areas might contribute to the faster low spatial frequency segmentation of a scene into salient regions (see also Knebel & Murray, Reference Knebel and Murray2012).
Might contour interpolation occur in a visual region lower than V4? Lower-to-middle-tier visual areas have also been implicated in figure contour interpolation, but their precise roles remain obscure. Traditional models of figure processing, inspired by single-unit studies of V1 and V2 response properties, relied on a feed-forward progression from striate to extrastriate cortex, to increasingly higher regions in lateral cortex. Yet full-scalp VEP experiments that explicitly compare the response properties of medial versus LO channels show that contour-specific responses in LO channels precede those in medial or dorsomedial channels (e.g., Halgren et al., Reference Halgren, Mendola, Chong and Dale2003; Shpaner et al., Reference Shpaner, Murray and Foxe2009; Knebel & Murray, Reference Knebel and Murray2012). So, although cue-invariant sensitivity to figure contours has been observed in V2, V3, V3a, and V3b in monkeys (von der Heydt & Peterhans, Reference Von Der Heydt and Peterhans1989; Mysore et al., Reference Mysore, Vogels, Raiguel and Orban2006; Sheth et al., Reference Sheth, Sharma, Rao and Sur1996; Leventhal et al., Reference Leventhal, Wang, Schmolesky and Zhou1998; Zeki et al., Reference Zeki, Perry and Bartels2003; Peterhans et al., Reference Peterhans, Heider and Baumann2005; Zhan & Baker, Reference Zhan and Baker2006) and humans (Dupont et al., Reference Dupont, DeBruyn, Vandenberghe, Rosier, Michiels, Marchal and Orban1997; van Oostende et al., Reference Van Oostende, Sunaert, Van Hecke, Marchal and Orban1997; Kastner et al., Reference Kastner, De Weerd and Ungerleider2000; Skiera et al., Reference Skiera, Petersen, Skalej and Fahle2000; Zeki et al., Reference Zeki, Perry and Bartels2003; Schira et al., Reference Schira, Fahle, Donner, Kraft and Brandt2004), this may not reflect contour interpolation per se but could be related to another cue-invariant process, such as scene segmentation.
Indeed, there is evidence to suggest indirectly that MT+ and regions of the LO complex might be driving such cue-invariant sensitivity to figures in lower visual cortex. For instance, Welchman et al. (Reference Welchman, Deubelius, Conrad, Bülthoff and Kourtzi2005) demonstrated that BOLD responses in both hMT+ and the LO complex (but not lower-tier areas) respond more strongly to the combination of two different cues to the depth structure (in this case, binocular disparity and linear perspective) than to either cue presented in isolation. Additionally, Chandrasekaran et al. (Reference Chandrasekaran, Canon, Dahmen, Kourtzi and Welchman2007) found that the same regions show BOLD response increases to the amount of global structural coherence in disparity-defined objects. Perhaps most intriguingly, fMRI studies by Ferber et al. (Reference Ferber, Humphrey and Vilis2003, Reference Ferber, Humphrey and Vilis2005) compared the duration of participants’ perceptual persistence of visually presented figure stimuli (i.e., subjective report of percept duration upon removal of the stimuli) with BOLD response duration. The authors found that perceptual persistence was linked to increased activity in LO cortex (and not hMT+), across different figure cues types. Wong et al. (Reference Wong, Aldcroft, Large, Culham and Vilis2009) found a similar link between perceptual persistence of objects and BOLD response duration in the LO complex as well as lower areas such as V2, V3, and V4. However, a brief (50 ms) duration flash of the object stimulus was linked only to the LO complex. Because the temporal profile of responses in the LO complex corresponded with briefly flashed as well as persistent stimuli, the authors concluded that regions in the complex are responsible for figure interpolation, and recruit the lower- and middle-tier regions for contour salience and persistence of the object percept.
Although the full picture is far from clear, these studies collectively support the notion that it is the higher-tier ventral (and possibly dorsal) regions that are responsible for figure contour interpolation, partially through their influence on lower regions of visual cortex. However, beyond this conclusion, there is no one region that can be considered without a doubt as being responsible for contour interpolation. More direct comparisons of areas such as V4, MT, and LO cortex will be necessary to better understand their respective contributions to the figure processing circuit. One aspect that may shed some light in this respect is the progression from two-dimensional figure images to volumetric surface structures, which will be discussed below.
2D shape sensitivity
Studies that explore sensitivity to object features typically employ some type of shape or feature categorization task, and this largely engages areas in the ventral visual stream (Lehky & Sereno, Reference Lehky and Sereno2007; Milner & Goodale, Reference Milner and Goodale2008). Within this cortical stream, sensitivity to shape has been observed to be roughly hierarchical (see Hegde & van Essen, Reference Hegdé and Van Essen2007), with V4 and IT showing progressively more sensitivity to feature complexity than area V2 (Desimone et al., Reference Desimone, Schein, Moran and Ungerleider1985; Kobatake & Tanaka, Reference Kobatake and Tanaka1994; Rust & DiCarlo, Reference Rust and Dicarlo2010; Carlson et al., 2011).
Aside from being implicated in top-down contour interpolation processes, V4 is particularly important for 2D shape classification. Activity in V4 has been linked to shape discrimination in humans (Vinberg & Grill-Spector, Reference Vinberg and Grill-Spector2008; Ales et al., Reference Ales, Appelbaum, Cottereau and Norcia2012). Additionally, Denys et al. (Reference Denys, Vanduffel, Fize, Nelissen, Peuskens, Van Essen and Orban2004) directly compared BOLD responses between monkeys and humans and found similar shape-selective response sensitivity in V4. How exactly does V4 contribute to shape processing? Single-unit recording studies on monkeys (Pasupathy & Connor, Reference Pasupathy and Connor2001, Reference Pasupathy and Connor2002; Rust & DiCarlo, Reference Rust and Dicarlo2010; Mysore et al., Reference Mysore, Vogels, Raiguel and Orban2006; Carlson et al., Reference Carlson, Rasquinha, Zhang and Connor2011) indicate that individual cells in V4 respond preferentially to particular contour curvatures, with most cells exhibiting an orientation-specific tuning for either convex or concave curvature that is invariant across retinal position. Pasupathy and Connor (Reference Pasupathy and Connor2002) have demonstrated that the orientation-specific curvature tuning of the individual cells likely represents a preference for object shape components and that the population responses of V4 can code for different shapes because of these response preferences. Because of this, V4 is regarded as a plausible mechanism for 2D shape coding.
Proceeding beyond V4, the human LO complex shows robust sensitivity to shape information (Malach et al., Reference Malach, Reppas, Benson, Kwong, Jiang, Kennedy, Ledden, Brady, Rosen and Tootell1995; Kanwisher et al., Reference Kanwisher, Chun, McDermott and Ledden1996; Altmann et al., Reference Altmann, Deubelius and Kourtzi2004; Vinberg & Grill-Spector, Reference Vinberg and Grill-Spector2008). In an attempt to localize areas of the brain with sensitivity to object shape, Grill-Spector et al. (Reference Grill-Spector, Kushnir, Edelman, Itzchak and Malach1998) parametrically varied the spatial extent of phase-scrambling of object stimuli, and found that the LO complex was more selective for coherent object images than V4, which demonstrated a more intermediate sensitivity. This implies that, while V4 is important for object shape processing, the higher regions of ventral visual cortex encode the most global, orientation and position-invariant object shape information. How do the different regions within the LO complex contribute to shape processing? Lesions in posterior parts of the complex such as LOR have been linked to deficits in shape categorization tasks (James et al., Reference James, Culham, Humphrey, Milner and Goodale2003). This region exhibits limited position, and size invariance, in contrast to the more invariant-shape selectivity of more anterior parts, including IT (Denys et al., Reference Denys, Vanduffel, Fize, Nelissen, Peuskens, Van Essen and Orban2004; Sawamura et al., Reference Sawamura, Georgieva, Vogels, Vanduffel and Orban2005; Stanley & Rubin, Reference Stanley and Rubin2005; Larsson & Heeger, Reference Larsson and Heeger2006). IT’s robust and rotation-invariant sensitivity to 2D shapes has been well established via monkey single-unit studies (Ito et al., Reference Ito, Tamura, Fujita and Tanaka1995; Fujita, Reference Fujita2003; Brincat & Connor, Reference Brincat and Connor2004; Lehky & Sereno, Reference Lehky and Sereno2007). In fact, Baylis and Driver (Reference Baylis and Driver2001) demonstrated that IT cells responded invariantly across luminance changes and mirror reversals of figure shapes, yet the cells were sensitive to stimuli that reversed in figure-ground arrangement, even though the contours of the stimuli remained unchanged. This offers convincing evidence that IT reflects a higher-level processing of object-based information across orientations and deformations than the more intermediate V4.
However, while shape processing is commonly thought to proceed hierarchically from V4 to areas of the LO complex, the actual dynamic network and its links to perception and task seem to be more complicated and variable. For instance, a VEP study employing cortical source localization methods (Ales et al., Reference Ales, Appelbaum, Cottereau and Norcia2012) reported results that suggest a reverse order of processing. In their study, activity time locked to a shape discrimination task was first observed in the LO complex and then in V4. While this particular sequence of activation may be limited to tasks that require top-down strategies for performance, the results nonetheless serve to emphasize the point that cortical processing of figure and object perception is only roughly hierarchical and may vary greatly according to task. A challenge to the field, however, is to understand why some monkey studies have reported that IT lesions did not significantly affect object shape discrimination (Huxlin et al., Reference Huxlin, Saunders, Marchionini, Pham and Merigan2000; Merigan & Saunders, Reference Merigan and Saunders2004), although V4 lesions have resulted in considerable deficits (Walsh et al., Reference Walsh, Butler, Carden and Kulikowski1992; Merigan & Pham, Reference Merigan and Pham1998; Girard et al., Reference Girard, Lomber and Bullier2002).
Aside from the ventral stream regions discussed above, some dorsal stream regions have been consistently implicated in shape discrimination tasks, particularly V3A and parts of the IPS (Denys et al., Reference Denys, Vanduffel, Fize, Nelissen, Peuskens, Van Essen and Orban2004; Lehky & Sereno, Reference Lehky and Sereno2007; Janssen et al., Reference Janssen, Srivastava, Ombelet and Orban2008; Klaver et al., Reference Klaver, Lichtensteiger, Bucher, Dietrich, Loenneker and Martin2008). Lehky and Sereno (Reference Lehky and Sereno2007) point out that dorsal stream shape sensitivity is much broader and weaker than regions such as IT and is likely not dependent upon input from the ventral stream. The role of these dorsal regions in object-related perception is controversial and will be discussed more extensively in the Discussion section.
Volumetric surface interpolation
In natural scenes, objects only very rarely cast perfectly symmetrical two-dimensional images on the retina. Instead their shapes can be more complex, they can be oriented in depth and can be obscured or partially hidden by other objects (known as “occlusion”). Thus, the visual system must interpolate coherent three-dimensional surfaces from the elements of a scene. This includes the binding of separate 2D image components into a surface and also completing surface edges that are hidden from view (sometimes both at once, see Fig. 6A).
Our own images of pictorially implied volume, inspired directly by Tse (Reference Tse1999). (A) Depicts a “sea monster” surface that requires the binding of multiple 2D components and the amodal completion of the monster’s hidden parts. (B) Depicts a self-occluding object surface.
An interesting question regarding the completion of occluded surfaces (amodal completion) is to what extent such completion differs in processing from the modal completion of illusory occluding surfaces like the Kanizsa triangle. Some researchers regard them as similar processes with shared mechanisms (e.g., Weil & Rees, Reference Weil and Rees2011), yet others think of them as distinct (Singh, Reference Singh2004; Zhou et al., Reference Zhou, Tjan, Zhou and Liu2008). Indeed, there are some indications that amodal completion may occur at a later processing stage than modal completion. For instance, human VEP experiments show that while the earliest responses among LO channels are similar for modal and amodal completion cues, negative potentials following a latency of about 250 ms are stronger and more sustained for amodal completion cues (Murray et al., Reference Murray, Foxe, Javitt and Foxe2004). Similar late negativity differences were observed by Brodeur et al. (Reference Brodeur, Lepore, Bacon and Debruille2009), who found that the later responses were higher for amodal completion cues than modal completion cues. The authors argued that these later responses are inversely weighted to the perceptual saliency of the stimulus and that amodal completion requires more attentional processing to compensate for the ambiguity of surface cues. Moreover, when a stimulus with both modal and amodal cues was used to explore the competition of the two cue types, the perceptual difficulty-related negative evoked responses were largest for amodal targets with modal cue distractors (Brodeur et al., Reference Brodeur, Lepore, Bacon and Debruille2009).
Thus, the comparison of modal and amodal completions suggests not only that the underlying interpolation processes may be different but also highlight the perceptual dominance of modally completed contours. After all, many experiments use the traditional Kanizsa images to investigate modal completion of the illusory “square” or “triangle” contours, rather than the amodally completed “circle” components. This may be because modal completion cues provide information about occluding objects, which in natural scenes are fully visible and closer to the observer than occluded objects. However, it may also be due to the perceptual ambiguity of objects that are partially hidden from view.
Our consideration of amodal completion, which concerns the interpolation of occluded surfaces, raises some more general questions concerning the processing of objects in depth. Does the perception of three-dimensional objects including complex self-occluding objects (Fig. 6B) as coherent volumetric surfaces simply depends on a hierarchical progression from 2D image sensitivity to 3D viewpoint-independent object recognition? Or is it more helpful to view all object interpolation processes in the brain as being biased toward volumetric surfaces, despite some observed sensitivity to unrealistic and impoverished 2D images used in experiments? Tse (Reference Tse1999) seems to argue for the latter interpretation, though the former is the more popular interpretation (see Kellman et al., Reference Kellman, Garrigan and Shipley2005). While more research is needed to clarify the issue, the truth may exist somewhere in between a strict hierarchy from 2D to 3D versus an obligatory 3D analysis that applies to all images.
As mentioned earlier, real objects provide multiple overlapping cues, including cues to the three-dimensional structure such as shading and binocular disparity. Moreover, binocular disparity tuning is apparent already at the level of V1 (Poggio & Fischer, Reference Poggio and Fischer1977; see also DeAngelis, Reference DeAngelis2000; Orban et al., Reference Orban, Janssen and Vogels2006; Roe et al., Reference Roe, Parker, Born and DeAngelis2007). Additional sensitivities are apparent in V2 and in V4, for example, a response preference for crossed disparities, indicating a bias for close objects (Hinkle & Connor, Reference Hinkle and Connor2002, Reference Hinkle and Connor2005). But are there cortical regions that code explicitly for a three-dimensional structure? The answer is yes, although the underlying processing may not reflect a classic hierarchical progression. It has been demonstrated that the brain regions involved in the processing of the 3D structure depend greatly upon the tasks employed to study the activity (e.g., Culham et al., Reference Culham, Danckert, DeSouza, Gati, Menon and Goodale2003; Valyear et al., Reference Valyear, Culham, Sharif, Westwood and Goodale2006; Cavina-Pratesi et al., Reference Cavina-Pratesi, Goodale and Culham2007). For this reason, we will discuss, in separate sections, the brain regions implicated in 3D shape classification, requiring viewpoint-independent object sensitivity; and those involved in the computation of changing 3D scene structure, requiring viewpoint-dependent sensitivity.
3D shape classification
As mentioned earlier, the ventral visual system exhibits a roughly hierarchical progression of sensitivity to increasingly more complex object shapes. While sensitivity to image complexity is important, object features are not limited to the two-dimensional images they cast upon the retina. On the contrary, real objects must be recognized despite changes of the retinal image due to manipulations such as rotation in depth. Experiments concerned with contour interpolation and basic shape sensitivity typically rely on simple 2D figure stimuli, but it is important to establish which regions of the cortex are merely sensitive to the shape of 2D retinal images, and which show viewpoint-independent sensitivity to object surface features, and are thus impervious to 3D transformations.
A few studies have directly compared 2D and 3D object cues. Regarding human subjects, Kourtzi et al. (Reference Kourtzi, Erb, Grodd and Bülthoff2003) investigated fMRI adaptation responses in the LO complex to changes of 2D and 3D features. The authors reported, overall, that the LO complex did not demonstrate adaptation effects when objects had the same 2D contour shape but different 3D shapes (concave vs. convex in depth)—indicating a particular sensitivity to changes in the 3D structure rather than 2D image features. Moreover, there was a response adaptation when objects had similar 3D shapes but different 2D-outlined shapes (due to 3D rotation)—supporting the notion that the LO complex is primarily tuned to volumetric rather than 2D surface cues. Although the complex was found to collectively demonstrate preference for 3D structural cues, the authors subsequently localized response sensitivity to changes in 3D shape (convexity vs. concavity in depth) to anterior regions of the complex such as posterior parts of IT and fusiform gyrus, noting that posterior regions such as the lateral occipital region (LOR) did not exhibit such sensitivity to 3D shape change. Thus, although LOR does show sensitivity to depth cues, it is perhaps limited to image rather than object features.
Similar to the anterior LO complex, single-unit studies on monkey IT (Janssen et al., Reference Janssen, Srivastava, Ombelet and Orban2008; Yamane et al., Reference Yamane, Carlson, Bowman, Wang and Connor2008; Hung et al., Reference Hung, Carlson and Connor2012; Verhoef et al., Reference Verhoef, Vogels and Janssen2012) show a sensitivity to 3D rather than 2D transformations, suggesting that the cells in the area respond to “inferred” 3D object cues from 2D displays rather than retinal image features. Moreover, somewhat resembling cells in V4 that exhibit differences in their preferred 2D contour curvatures (Pasupathy & Connor, Reference Pasupathy and Connor2001; Carlson et al., Reference Carlson, Rasquinha, Zhang and Connor2011), IT cells have been shown to differ in their preference for volumetric shape; these differences allow for the interpolation of a coherent viewpoint-independent representation of the object structure via the combination of cell responses to their respective preferred component shapes (Brincat & Connor, Reference Brincat and Connor2004; Yamane et al., Reference Yamane, Carlson, Bowman, Wang and Connor2008; Hung et al., Reference Hung, Carlson and Connor2012). These results seem compatible with a classic largely hierarchical theory for 3D shape processing such as that of Marr and Nishihara (Reference Marr and Nishihara1978), according to which object structures are perceptually derived from combinations of encoded structure components or “geons” (see also Beiderman, Reference Biederman2001).
Thus, hierarchical progression of invariant shape sensitivity is generally observed in the ventral visual stream for object categorization tasks, although the progression of sensitivity may be more of a graded one than a discrete step-wise progression (Hegde & van Essen, Reference Hegdé and Van Essen2007; Rust & DiCarlo, Reference Rust and Dicarlo2010). IT perhaps represents a point in the processing hierarchy where object, rather than image, features are coded. This ultimately allows for the differentiation and generalization of object stimuli by categories such as “Animate” versus “Inanimate,” or “Face” versus “Body”; an organizing principle that is similarly reflected in human fMRI and monkey single-unit recording studies (Tanaka, Reference Tanaka2003; Kriegskorte et al., Reference Kriegeskorte, Mur, Ruff, Kiani, Bodurka, Esteky, Tanaka and Bandettini2008; Haxby et al., Reference Haxby, Guntupalli, Connolly, Halchenko, Conroy, Gobbini, Hanke and Ramadge2011). Indeed, the number and structure of category representations remain a provocative question (e.g., Huth et al., Reference Huth, Nishimoto, Vu and Gallant2012) as activity in regions such as the fusiform face area (FFA) is believed to instantiate the differentiation of exemplars within a shared category, most prominently demonstrated for face discrimination (Kanwisher et al., Reference Kanwisher, McDermott and Chun1997; Grill-Spector & Malach, Reference Grill-Spector and Malach2004; Loffler et al., Reference Loffler, Yourganov, Wilkinson and Wilson2005), but perhaps also for any fine-level within-category discrimination that is driven by training and expertise (Gauthier et al., Reference Gauthier, Tarr, Anderson, Skudlarski and Gore1999, Reference Gauthier, Skudlarski, Gore and Anderson2000).
Yet this ventral circuit is not the only means in which the brain encodes the object structure. The Global 3D structure section discusses areas in the dorsal visual stream that calculate changes in the global depth structure of a visual scene, which is important for object tracking and manipulation rather than categorization.
Global 3D structure
While viewpoint-independent feature sensitivity is important for tasks related to categorization and recognition, sensitivity to image features that are viewpoint-dependent is important for tracking the spatial position and orientation of objects in relation to one another, as well as to the observer’s own egocentric space. Such information is crucial for tasks requiring navigation, object pursuit, and object manipulation (Valyear et al., Reference Valyear, Culham, Sharif, Westwood and Goodale2006; Milner & Goodale, Reference Milner and Goodale2008). Unlike object categorization, these types of tasks are primarily associated with activity in the dorsal visual stream (Haxby et al., Reference Haxby, Grady, Horwitz, Ungerleider, Mishkin, Carson, Herscovitch, Schapiro and Rapoport1991; Milner & Goodale, Reference Milner and Goodale2008). The overall function of the dorsal stream is somewhat controversial, as will be addressed later. However, we propose that dorsal regions sensitive to figure and object stimuli are important for processing the changing the depth structure of a scene, thereby allowing for viewpoint-dependent object sensitivity. Orban (Reference Orban2011) defines depth structure as “the variation of depth along the surface or edge of an object.” Thus, object-sensitive regions in the dorsal stream process information about the three-dimensional surface layout of an observer’s surroundings, particularly the continuous changes that arise from the object and observer movement (Borst et al., Reference Borst, Thompson and Kosslyn2011).
Because Mishkin et al. (Reference Mishkin, Ungerleider and Macko1983) famously observed that large lesions along the dorsal stream were found to affect a monkey’s ability to detect an object’s position in space, the stream is often thought to be necessary for spatial vision, including depth. However, depth processing is much more complicated than the classic dichotomy suggests, and it involves both visual streams. For instance, Neri et al. (Reference Neri, Bridge and Heeger2004) used fMRI to compare activity in dorsal and ventral regions to absolute and relative binocular disparity. Relative disparity—or the angular difference in binocular disparity between two objects—can provide cues to the respective depths of multiple objects. The authors concluded that ventral stream regions were more sensitive to relative disparity information than dorsal stream regions, which appeared as particularly sensitive to absolute disparities. However, several subsequent studies indicate that regions in the dorsal stream do indeed process relative disparity. For instance, Cottereau et al. (Reference Cottereau, McKee, Ales and Norcia2011, also see Reference Cottereau, McKee, Ales and Norcia2012a, Reference Cottereau, McKee and Norcia2012b) observed that human V3A was particularly sensitive to relative disparity defined by a center-surround configuration of dot stimuli with distinct binocular disparities for center versus surround. In fact, the gain modulation of the V3A response was found to be proportional to the disparity difference. This metric sensitivity to relative disparity was also found in V3 and V3A of macaque via single-unit and multiunit recordings (Anzai et al., Reference Anzai, Chowdhury and DeAngelis2011). A separate single-unit study on monkeys by Krug and Parker (Reference Krug and Parker2011) varied relative disparity values of transparent overlapping “planes” of moving dots and found that cells in MT+ are tuned to relative, as well as absolute, disparity. What might account for the discrepancy of findings between these studies and Neri et al. (Reference Neri, Bridge and Heeger2004)?
As mentioned in the Global 3D structure section, the ventral stream is largely associated with stimulus classification, and a look at the tasks employed in the above studies seems to support this interpretation. While Anzai et al. (Reference Anzai, Chowdhury and DeAngelis2011) and Cottereau et al. (Reference Cottereau, McKee, Ales and Norcia2011) both relied solely on passive subject fixation, Neri et al. (Reference Neri, Bridge and Heeger2004) employed a depth discrimination task. Krug and Parker (Reference Krug and Parker2011) interleaved a similar discrimination task within their viewing protocol to assess the monkeys’ performance, but the spiking data reported were taken only from the passive viewing trials. Moreover, a single-unit study on monkeys by Shiozaki et al. (Reference Shiozaki, Tanabe, Doi and Fujita2012) linked the response thresholds of V4 cells for relative disparity to the monkeys’ discrimination thresholds in a psychophysical task. Collectively, these results are compatible with our interpretation of ventral stream regions as being linked to tasks involving classification. While the ventral stream has been shown to respond to relative disparity information, it may be primarily limited to categorical judgments of relative depth such as a disparity discrimination task. In an fMRI study by Preston et al. (Reference Preston, Li, Kourtzi and Welchman2008), the authors endorse a similar conclusion. In the study, figures of varying disparities were presented to subjects as they performed an unrelated task to ensure attentive viewing. The authors found that dorsal areas such as V3A, V3B, V7, and intraparietal sulcus were found to exhibit a metric sensitivity to absolute disparity, while regions of the LO complex instead showed categorical sensitivity to “depth sign,” in which a disparity-defined plane was either “behind” or “in front of” the fixation point. The study focused on absolute rather than relative disparity cues, but the results suggest that binocular disparity as processed in the ventral stream may have less to do with depth computation per se and more to do with qualitative object attributes. This is consistent with the interpretation of ventral stream processing as being primarily for viewpoint-independent categorization.
Thus, in contrast to the ventral stream, dorsal stream areas seem to show a metric sensitivity to changes in scenic depth structure, which includes figures defined against a background. Although midlevel dorsal regions such as V3A and V3B have previously been interpreted as specifically important for figure contour interpolation (Dupont et al., Reference Dupont, DeBruyn, Vandenberghe, Rosier, Michiels, Marchal and Orban1997; van Oostende et al., Reference Van Oostende, Sunaert, Van Hecke, Marchal and Orban1997; Zeki et al., Reference Zeki, Perry and Bartels2003; Schira et al., Reference Schira, Fahle, Donner, Kraft and Brandt2004), studies that have explicitly compared responses in these regions across different depth cues indicate instead that they are particularly sensitive to cues to 3D structure (Beer et al., Reference Beer, Blakemore, Previc and Liotti2002; Tyler et al., Reference Tyler, Likova, Kontsevich and Wade2006; Preston et al., Reference Preston, Kourtzi and Welchman2009; Ban et al., Reference Ban, Preston, Meeson and Welchman2012). Ban et al. (Reference Ban, Preston, Meeson and Welchman2012) in particular highlight the distinct sensitivity of V3B to combined depth structure cues (motion contrast and disparity), specifically when the cues offer compatible indexes of structure in depth. In this light, we propose that the metric sensitivity of dorsal areas such as V3A and V3B to disparity and motion contrast information (e.g., Preston et al., Reference Preston, Li, Kourtzi and Welchman2008; Anzai et al., Reference Anzai, Chowdhury and DeAngelis2011; Fesi et al., Reference Fesi, Yannes, Brinckmann, Ales, Norcia and Gilmore2011; Cottereau et al., Reference Cottereau, McKee, Ales and Norcia2011) should be thought to reflect sensitivity to scene structure cues. Such a metric sensitivity to the changing global depth structure of a scene is consistent with the interpretation of the dorsal stream as generally important for viewpoint-dependent object processing (Gauthier et al., Reference Gauthier, Hayward, Tarr, Anderson, Skudlarski and Gore2002; James et al., Reference James, Humphrey, Gati, Menon and Goodale2002; Vanrie et al., Reference Vanrie, Béatse, Wagemans, Sunaert and Van Hecke2002; Kim & Biederman, Reference Kim and Biederman2012). This type of information would allow for the visual tracking of object features in depth (Caplovitz & Tse, Reference Caplovitz and Tse2007), as well as observer–object interactions such as grasping (Culham et al., Reference Culham, Danckert, DeSouza, Gati, Menon and Goodale2003; Valyear et al., Reference Valyear, Culham, Sharif, Westwood and Goodale2006).
Discussion
The experiments reviewed above offer important insight into the neural mechanisms of figure perception, either by directly exploring linking hypotheses between physiological and psychophysical data or by allowing us to construct less specific links by analogy. In this article, a processing model was proposed to provide a contextual framework for the experiments, with stages that proceed hierarchically according to function. The earliest stage of cortical processing is one of the best established in vision research, and concerns basic visual features. The second stage represents the segmentation of salient regions in a scene. This is followed by the interpolation of a figure’s contours, which allows for the identification of the figure’s shape. The fifth stage concerns volumetric surface interpolation, which allows for subsequent shape classification of three-dimensional surfaces, and the tracking of objects as they, as well as the observer, move in depth.
Our model and the surrounding literature review clearly serve as robust examples of the brain/behavior correlation claims that Teller (Reference Teller1984) sought to evaluate via families of linking propositions. We note in passing that several of the propositions and their implications are manifest here. In particular, our emphasis on neural responses that are invariant to cue type (contour interpolation), size, position, and rotation (2D shape sensitivity), or rotation in depth (3D shape classification) all represent strategic information loss that leads to a mapping between stimuli and brain states that is many:1, within the Identity family. Second, implicit in our discussion of metric versus categorical neural responses for contour strength or for 3D depth is the idea of a Similarity family in which continuous “monotonic variation in neural activity implies a monotonic variation in sensation and vise versa.” Third, Teller cited bistable figures (and in our current context, the associated border ownership) as an example of the Mutual Exclusivity family of propositions in which percepts would have mutually exclusive brain states. Finally, her concept of the “bridge locus”—the set of neurons whose activities form the immediate substrate of (conscious) visual perception—is as relevant as ever. Teller’s ideas seem to us to often be testable in principle and might motivate future experiments.
The current consensus regarding the neural substrates of our model’s processing stages can be mapped out very roughly in terms of cortical regions, but the precise details of the underlying circuits remain unknown. For instance, the point in the circuit at which feed-forward processing of object information first cues top-down processes via feedback connections has yet to be clearly determined and is probably task dependent. As discussed above, this likely happens in lateral regions of occipital cortex for contour interpolation, yet the extent to which a midlevel area such as V4, as opposed to a somewhat higher region such as LOR, contributes to contour interpolation and 2D shape processing is not clear. Dissociating the functions of these regions will therefore be essential to understanding the dynamics of the circuit, especially the transition from 2D to 3D feature processing.
Generalizing beyond specific cortical regions, it may be important to understand figure processing in the larger context of the dorsal and ventral visual streams. These are thought to be functionally dissociable, with the ventral stream conventionally interpreted as important for the categorization and discrimination of the permanent features of objects, irrespective of the observer’s viewpoint, whereas the dorsal stream is thought to instantiate the visual control of an observer’s actions and is therefore more sensitive to changes related to an observer’s viewpoint of objects (Mishkin et al., Reference Mishkin, Ungerleider and Macko1983; Haxby et al., Reference Haxby, Grady, Horwitz, Ungerleider, Mishkin, Carson, Herscovitch, Schapiro and Rapoport1991; Kobatake & Tanaka, Reference Kobatake and Tanaka1994; Passingham & Toni, Reference Passingham and Toni2001; James et al., Reference James, Humphrey, Gati, Menon and Goodale2002; Sereno et al., Reference Sereno, Trinath, Augath and Logothetis2002; Vanrie et al., Reference Vanrie, Béatse, Wagemans, Sunaert and Van Hecke2002; Cox et al., Reference Cox, Meier, Oertelt and DiCarlo2005; Valyear et al., Reference Valyear, Culham, Sharif, Westwood and Goodale2006; Milner & Goodale, Reference Milner and Goodale2008). Studies that highlight the categorical response properties of ventral areas versus the metric response properties of dorsal areas (Preston et al., Reference Preston, Li, Kourtzi and Welchman2008; Thielscher et al., Reference Thielscher, Kölle, Neumann, Spitzer and Grön2008; Fesi et al., Reference Fesi, Yannes, Brinckmann, Ales, Norcia and Gilmore2011; Altschuler et al., Reference Altschuler, Molholm, Russo, Snyder, Brandwein, Blanco and Foxe2012; Cottereau et al., Reference Cottereau, McKee, Ales and Norcia2011) can be seen to support such a functional dissociation of the two processing streams.
Despite the usefulness of the two-stream model, there are researchers who argue that a conceptualization of the dorsal and ventral streams as functionally dichotomous—be it “what versus where,” “what versus how,” or “action versus perception,”—to be overly simplistic, obscuring important details of the visual system. For instance, in their review of functional and anatomical evidence from monkey cortex, Kravitz et al. (Reference Kravitz, Saleem, Baker and Mishkin2011) argued that there are at least three pathways within the dorsal stream—the parieto-middle temporal, parieto-prefrontal, and parieto-premotor pathways—each important for distinct functions. Schenk and McIntosh (Reference Schenk and McIntosh2010) stressed the extensive connections and interactions between the dorsal and ventral streams. Similarly, both dorsal hMT and ventral LO complex have been implicated in top-down processing related to figure contours (Kourtzi & Huberle, Reference Kourtzi and Huberle2005; Likova & Tyler, Reference Likova and Tyler2008) and have shown similar response sensitivity to object coherence (Chandrasekaran et al., Reference Chandrasekaran, Canon, Dahmen, Kourtzi and Welchman2007) and the combination of depth cues (Welchman et al., Reference Welchman, Deubelius, Conrad, Bülthoff and Kourtzi2005).
One unresolved controversy that is pertinent to linking hypotheses for figure processing concerns this dorsal stream and its contribution to conscious visual perception. The data primarily come from experiments using a method of dichoptic image presentation known as “continuous flash suppression.” In this paradigm, an object stimulus is presented to a participant’s nondominant eye, and a distractor grating is presented to the dominant eye, resulting in prolonged periods during which the participant reports no percept of the object. Despite the lack of object awareness, activity in the dorsal stream shows a robust object response, indicating that it is dissociated from conscious object perception (Fang & He, Reference Fang and He2005; Almeida et al., Reference Almeida, Mahon, Nakayama and Caramazza2008; Sakuraba et al., Reference Sakuraba, Sakai, Yamanaka, Yokosawa and Hirayama2012). However, others argue that by using an objective assessment of perceptual sensitivity, such as the two-alternative forced choice (2AFC) method, rather than subjective reports, a neural link to conscious object perception is revealed that is similar for both ventral and dorsal stream regions (Hesselmann & Malach, Reference Hesselmann and Malach2011; Hesselmann et al., Reference Hesselmann, Hebart and Malach2011). Regardless, data from multiple lesioning studies has lead Milner (Reference Milner2012) to conclude that, although activity in dorsal regions can sometimes be linked to tasks related to conscious perception, perhaps due to connections to other areas in the brain, the stream itself has limited access to conscious perception.
As was the case for our description of the automatic segmentation of scenes into salient regions, an unconscious process could still be regarded as an important influence on our behavior (Kimchi & Peterson, Reference Kimchi and Peterson2008). Indeed, in general, human behavior is no doubt affected strongly by many unconscious influences, in which neural activity is impactful but is not reflected in the contents of awareness. However, typically, linking hypotheses are intended to formalize the neural activity that is directly positively (or negatively) correlated with a conscious behavioral threshold or outcome. Therefore, in the case of the proposed dorsal stream dissocation from object-related awareness, it is important to consider both the presence and the absence of specific brain/behavior correlations in order to form a comprehensive understanding of how a region or circuit of the brain contributes to conscious visual perception (i.e., the bridge locus). A second implication of Milner’s (Reference Milner2012) conclusion is that linked relationships of neural activity and behavior depend greatly upon the methods and tasks employed and may not necessarily reflect a constant overall function of a region or network.
Indeed, this controversy concerning the dorsal stream points to the more general issue of how individual linking relationships are constrained by the tasks and recording methods used to explore them. Brain–behavior links are often investigated within a limited range of tasks, such as onset detection or feature discrimination. It has been noted before that certain studies found no significant differences in responses for detection tasks and passive viewing conditions (Murray et al., Reference Murray, Wylie, Higgins, Javitt, Schroeder and Foxe2002; Bakar et al., Reference Bakar, Liu, Conci, Elliott and Ioannides2008; Guzzon & Casco, Reference Guzzon and Casco2011; Appelbaum et al., Reference Appelbaum, Ales and Norcia2012), yet the generalizability of the links across different tasks was not formally investigated. Studies that have directly compared links across tasks for similar stimuli include those that support a functional dissociation of the dorsal and ventral streams for grasping/tracking and categorization tasks, respectively (e.g., Culham et al., Reference Culham, Danckert, DeSouza, Gati, Menon and Goodale2003; Valyear et al., Reference Valyear, Culham, Sharif, Westwood and Goodale2006; Cavina-Pratesi et al., Reference Cavina-Pratesi, Goodale and Culham2007). For instance, Cavina-Pratesi et al. (Reference Cavina-Pratesi, Goodale and Culham2007) directly compared BOLD responses to real 3D objects viewed passively or presented for grasping and reaching tasks as well as size and pattern discrimination tasks. The authors found activity in both dorsal and ventral regions in the passive viewing condition. However, the tasks each yielded distinct activation patterns, with BOLD activity in intraparietal cortex being linked to grasping and reaching, and LOR being linked to the feature discrimination tasks. Thus, if we wish to obtain a comprehensive understanding of how the visual system contributes to figure perception, a thorough investigation of how linking relationships are constrained by task will be necessary.
Much has been discovered with respect to how the brain instantiates our perception of figures, and we look at the prospects for future insights into relevant linking propositions with great optimism. We already know of many regions of interest thought to be important for figure perception. Now it is important to understand the dynamics of the circuit underlying figure detection, classification, and tracking, especially how the processes may vary from task to task. Also, the ongoing exchange between human and invasive nonhuman experiments will be vital for understanding mechanisms at the single-neuron level. Finally, we hope to eventually be able to describe how this circuit changes with age and experience and explain how such changes reflect important differences of perception.
