Retinotopy in visual cortex
A fundamental principle of visual neuroscience is that receptive fields are organized based on retinal coordinates. Initially, the orderly arrangement of receptive fields based on retinal position creates a retinotopic map in early visual areas, such that adjacent neurons tend to respond to stimulation from adjacent areas of the retina. Already at the V1 level, however, the continuity rule is broken both for the upper and lower quadrant and more spectacularly for the left and right visual hemifields, where the link between the two hemifields is mainly mediated by the callosal projections.
Retinotopic coding can provide an effective organizing principle for the initial stages of visual processing. In particular, retinotopy could help the visual system to maintain coherent spatial binding of features, despite the distributed processing of different features across various cortical areas. For example, it has been shown that the processing of color and motion information involves distinct neural populations. Yet, under most circumstances, these visual features from anatomically separate “modules” are correctly bound together in our perception. This binding seems to be based, at least initially, on retinotopic coding, i.e., the spatiotemporal coincidence in retinal cortical maps (Melcher et al., Reference Melcher, Papathomas and Vidnyanszky2005; Freiwald, Reference Freiwald2007; Lin & He, Reference Lin and He2009). Thus, the widespread use of retinal coding throughout the visual hierarchy would be a good way to maintain a coherent spatial representation during specialized processing of simple and complex visual features in different brain regions.
Retinotopic coordinates prove less useful, however, under natural viewing conditions in which people move their eyes, head, and body. Each self-movement is likely to change the position of stimuli on the retina, creating the problem of visual stability (for review, see Melcher, Reference Melcher2011; Burr & Morrone, Reference Burr and Morrone2011; Wurtz, Reference Wurtz2008; Wurtz et al., Reference Wurtz, Joiner and Berman2011). The basic problem, as shown in Fig. 1, is that a saccadic eye movement creates a shift of the projected location of the object on the retina. In this case, the saccade would lead to the same external stimulus being processed by a new set of visual neurons, in the opposite cerebral hemisphere, after the saccade. At a subjective level, the fact that the world does indeed seem stable, except under certain laboratory conditions, suggests that perception does not depend entirely on retinal position. How a stable representation of the world is created from the output of mobile sensors is an old and venerable problem that has fascinated many scientists, including Descartes, Helmholtz, Mach and Sherrington, and indeed goes back to the 11th century Persian scholar Abū `Alī al-Hasan ibn al-Hasan ibn al-Haytham (Latinized “Alhazen”): (for review, see: Melcher & Colby, Reference Melcher and Colby2008; Wurtz, Reference Wurtz2008; Burr & Morrone, Reference Burr and Morrone2011; Melcher, Reference Melcher2011; Wurtz et al., Reference Wurtz, Joiner and Berman2011).
Illustration of the problem of visual stability as a result of saccades. A change in gaze position from the right side of an object to the left side of an object, via a horizontal saccade, would create a situation in which the neural processing of that object in visual areas would switch hemispheres, involving a completely new set of neurons. Nonetheless, our subjective perception of the object is that it remains stable in external space and identity across the gaze shift.
One might imagine that visual processing could essentially start fresh with each fixation (Dennett, Reference Dennett1991; Irwin, Reference Irwin1991). However, given that we perform 3–5 saccades per second, our brain would have very limited time to process the incoming information, and crucially we would throw away useful predictions about the state of the world at the beginning of each fixation. Information about the upcoming fixation is available from peripheral previewing and attention is shifted to the new saccadic target prior to saccade onset (Kowler et al., Reference Kowler, Anderson, Dosher and Blaser1995; Deubel & Schneider, Reference Deubel and Schneider1996). Thus, it would make sense to utilize this presaccadic information in postsaccadic processing. From the perspective of active perception, vision and the other senses involve sensorimotor sampling routines in which actions/saccades and perceptions are closely intertwined (for review, see: Schroeder et al., Reference Schroeder, Wilson, Radman, Scharfman and Lakatos2010). In support for this idea, a number of studies have shown that making eye movements modulates the spatial selectivity of the receptive fields of neurons in visual areas (see section 3, below) and most importantly the effects are present before the actual eye movements, suggesting the implementation of a predictive code.
Most studies of the functional responses of the visual system are based on the change in activity of a neuron when a particular stimulus is flashed in its retinotopically-defined receptive field during steady fixation. Thus, it is important to also investigate the receptive field properties of neurons under more natural viewing conditions in which the eyes, head, and body are free to move. In particular, this review article focuses on evidence for nonretinotopic coding when the object moves or the observer moves. Such studies are critical to resolve the seeming contradiction between retinotopic coding and our subjective experience of a stable world.
Behavioral evidence for nonretinotopic visual processing
The study of nonretinotopic processing has benefitted from converging evidence from behavioral, neurophysiological, and neuroimaging methods. In the case of behavior, a number of nonretinotopic effects have been found in studies of perception around the time of eye movements. One of the most compelling examples is the spatial mislocalization of briefly flashed stimuli in the time period before or after a saccade. A robust and consistent finding is that stimuli presented close to fixation within a time period of around 50–100 ms before saccade onset to around 50–100 ms after saccade offset are mislocalized in the direction of the saccade (Matin & Pearce, Reference Matin and Pearce1965; Bischof & Kramer, Reference Bischof and Kramer1968) or, when presented at locations around the saccade target, compressed toward the saccadic target (Morrone et al., Reference Morrone, Ross and Burr1997; Ross et al., Reference Ross, Morrone and Burr1997; Lappe et al., Reference Lappe, Awater and Krekelberg2000). The pattern of peri-saccadic mislocalization can be very complex and vary depending on the amount of relevant visual information (see Morrone, Reference Morrone and Chalupa2014). However, the fact that the errors are present before the actual movement, points again to a reorganization of the spatial selectivity of visual neurons orchestrated by a corollary discharge signal that informs vision of the upcoming saccade.
Another of the main findings in these behavioral studies is that perceptual reports are consistent with the transfer/integration of feature information across saccades. Examples include the perception of shape/orientation/form (Hayhoe et al., Reference Hayhoe, Lachter and Feldman1991; Melcher, Reference Melcher2005; Prime et al., Reference Prime, Niemeier and Crawford2006, Reference Prime, Vesia and Crawford2011; Melcher, Reference Melcher2007; Gordon et al., Reference Gordon, Vollmer and Frankl2008; Van Eccelpoel et al., Reference Van Eccelpoel, Germeys, De Graef and Verfaillie2008; Demeyer et al., Reference Demeyer, De Graef, Wagemans and Verfaillie2009, Reference Demeyer, De Graef, Wagemans and Verfaillie2010, Reference Demeyer, De Graef, Verfaillie and Wagemans2011; Fracasso et al., Reference Fracasso, Caramazza and Melcher2010; Zirnsak et al., Reference Zirnsak, Gerhards, Kiani, Lappe and Hamker2011; Zimmermann et al., Reference Zimmermann, Morrone, Fink and Burr2013a, Reference Zimmermann, Morrone and Burrb; Cha & Chong, Reference Cha and Chong2014), color (Wittenberg, Bremmer & Wachtler, Reference Wittenberg, Bremmer and Wachtler2008), motion (Melcher & Morrone, Reference Melcher and Morrone2003; Ong et al., Reference Ong, Hooshvar, Zhang and Bisley2009; Fracasso et al., Reference Fracasso, Caramazza and Melcher2010; Biber & Ilg, Reference Biber and Ilg2011; Melcher & Fracasso, Reference Melcher and Fracasso2012; Seidel Malkinson et al., Reference Seidel Malkinson, Mckyton and Zohary2012; Turi & Burr, Reference Turi and Burr2012) and perceived time duration (Burr et al., Reference Burr, Tozzi and Morrone2007). A key similarity of these different studies is that perceptual responses are based on combining information from different retinal locations, and this requires the alignment of the visual information in external (spatiotopic) coordinates or the alignment of the presaccadic and postsaccadic locations of an object.
A simple example of the spatiotopic perception of form and motion is the perception of coherent “transformational apparent motion” (Tse et al., Reference Tse, Cavanagh, Nakayama and Watanabe1998) across saccades (Fig. 2). The presentation of two frames showing different shapes, separated by a brief interstimulus interval (ISI), can be perceived as a coherent motion event in which the shape is transformed. In the example in Fig. 2, an oriented bar is shown in the first frame and then, after the saccade, two bars are presented: one at the same retinal position and one at the same spatial/screen position. Observers perceive the rotation of the bar in the spatiotopic, rather than the retinotopic, coordinate system (Fracasso et al., Reference Fracasso, Caramazza and Melcher2010).
Illustration of spatiotopic perception of apparent motion. Events in the trial are presented from top-left to bottom-right. Observers were presented with an oriented bar, which disappeared prior to the saccade onset. On the new fixation, two bars were presented, one in the same spatial position on the screen and the other in the same retinal position with respect to fixation. Participants reported the rotation of the bar, as a measure of whether the form was matched in spatiotopic or retinotopic coordinates. As shown in the panel inset, observers consistently reported rotation of the bar consistent with spatiotopic matching. Data previously reported in Fracasso et al. (Reference Fracasso, Caramazza and Melcher2010).
As shown in Figs. 1 and 2, the most straightforward (but certainly not the only) way to measure spatiotopic perceptual effects is to show two stimuli over time in the same spatial position, separated by a saccade which causes the two stimuli to fall in different retinal positions. Given that these two stimuli can interact perceptually in various ways, such as priming, integration, adaptation, and so on, the presence of such interactions can be used as evidence for nonretinotopic processing and in particular for spatiotopic effects. A large number of studies have reported spatiotopic effects (Hayhoe et al., Reference Hayhoe, Lachter and Feldman1991; Melcher & Morrone, Reference Melcher and Morrone2003; Melcher, Reference Melcher2005, Reference Melcher2007, Reference Melcher2009; Prime et al., Reference Prime, Niemeier and Crawford2006, Reference Prime, Vesia and Crawford2011; Burr et al., Reference Burr, Tozzi and Morrone2007; Ezzati et al., Reference Ezzati, Golzar and Afraz2008; Gordon et al., Reference Gordon, Vollmer and Frankl2008; Van Eccelpoel et al., Reference Van Eccelpoel, Germeys, De Graef and Verfaillie2008; Wittenberg et al., Reference Wittenberg, Bremmer and Wachtler2008; Demeyer et al., Reference Demeyer, De Graef, Wagemans and Verfaillie2009, Reference Demeyer, De Graef, Wagemans and Verfaillie2010, Reference Demeyer, De Graef, Verfaillie and Wagemans2011; Ong et al., Reference Ong, Hooshvar, Zhang and Bisley2009; Fracasso et al., Reference Fracasso, Caramazza and Melcher2010; Fracasso et al., Reference Fracasso, Caramazza and Melcher2010; Biber & Ilg, Reference Biber and Ilg2011; Burr et al., Reference Burr, Cicchini, Arrighi and Morrone2011; Au et al., Reference Au, Ono and Watanabe2012; Melcher & Fracasso, Reference Melcher and Fracasso2012; Seidel Malkinson et al., Reference Seidel Malkinson, Mckyton and Zohary2012; Turi & Burr, Reference Turi and Burr2012; Zimmermann et al., Reference Zimmermann, Morrone, Fink and Burr2013a, Reference Zimmermann, Morrone and Burrb; Cha & Chong, Reference Cha and Chong2014; Corbett & Melcher, Reference Corbett and Melcher2014; Jonikaitis & Belopolsky, Reference Jonikaitis and Belopolsky2014; Nakashima & Sugita, Reference Nakashima and Sugita2014), while other studies have found only retinotopic effects (Wenderoth & Wiese, Reference Wenderoth and Wiese2008; Afraz & Cavanagh, Reference Afraz and Cavanagh2009; Knapen et al., Reference Knapen, Rolfs and Cavanagh2009). As will be described below, this difference might result from various causes, including: the level of the cortical processing involved, the dynamics of the presentation of the stimuli, the attentional state of the subject, the memory load and the predictions and expectations of the subject.
One area of particular controversy is the reference frame of adaptation aftereffects. In a traditional adaptation paradigm, an adapting stimulus is shown for several seconds and then, after a delay, a probe stimulus is presented at the same location (Gibson, Reference Gibson1937; for review, see Webster, Reference Webster2011). Due to adaptation, the perceived features of the probe are “repulsed” in the opposite direction from the adapter (an adaptation aftereffect). Adaptation is a common visual phenomenon, reported for both relatively low-level features, such as contrast, and high-level features such as face identity or gender. In the case of tilt adaptation, for example, one study has reported that it is retinotopic (Knapen et al., Reference Knapen, Rolfs and Cavanagh2009), while others have reported nonretinotopic effects (Melcher, Reference Melcher2005; Zirnsak et al., Reference Zirnsak, Gerhards, Kiani, Lappe and Hamker2011; Zimmermann et al., Reference Zimmermann, Morrone, Fink and Burr2013a,Reference Zimmermann, Morrone and Burrb; Cha & Chong, Reference Cha and Chong2014; Nakashima & Sugita, Reference Nakashima and Sugita2014). More generally, it is important to point out that studies of the retinotopy or spatiotopy of perceptual effects have differed in terms of the features tested (contrast, orientation, shape, motion, face identity, and so on), the timing of the adapter and probe, the number of intervening saccades and the nature of the contingency between adapter and test. There are at least four factors that might help to explain the divergence of findings.
First, it has been argued that relatively low-level visual properties are retinotopic while more high-level properties show more of a spatiotopic effect (Melcher, Reference Melcher2005; Melcher & Colby, Reference Melcher and Colby2008). In the case of motion, for example, simple translational motion shows a retinotopic aftereffect, while the positional illusion induced by motion is spatiotopic (Turi & Burr, Reference Turi and Burr2012). Likewise, positive motion priming was found to be spatiotopic, while negative priming was predominantly retinotopic (Yoshimoto et al., Reference Yoshimoto, Uchida-Ota and Takeuchi2014a,Reference Yoshimoto, Uchida-Ota and Takeuchib), and whether positive or negative priming occurs depending on a range of factors such as duration, luminance contrast, and velocity of the stimulus used. One complication in any study of adaptation is to distinguish between low-level adaptation (local contrast or orientation) and high-level properties, as both might be adapted simultaneously. For instance, face adaptation can be based on local features such as shape and many paradigms, using morphing software to create stimuli, would confound changes in gender or identity with local shape changes (Dennett et al., Reference Dennett, Edwards and Mckone2012). Thus, studies aiming to study high-level adaptation effects might actually study adaptation at multiple levels of visual complexity.
Second, several studies suggest that nonretinotopic visual coordinates take time to update (Zimmermann et al., Reference Zimmermann, Morrone, Fink and Burr2013a,Reference Zimmermann, Morrone and Burrb). In one study of the tilt aftereffect, Zimmerman and colleagues varied the amount of preview time of the saccadic target prior to a visually driven saccade. With longer preview periods, there was increasing spatiotopic Tilt after-effect (TAE) and reduced retinotopic TAE, consistent with the idea that the spatiotopic representation was updated over a period of one second or longer. Similar ideas come from studies of the remapping of an attended location, in particular with inhibition of return (IOR) measurements. In the IOR paradigm, reaction times to an exogenous distractor presented in an irrelevant location become longer than RTs to a neutral, noncued location. Numerous studies have shown that IOR is spatiotopic (Sapir et al., Reference Sapir, Hayes, Henik, Danzinger and Rafal2004; Mathot & Theeuwes, Reference Mathot and Theeuwes2011; Pertzov et al., Reference Pertzov, Avidan and Zohary2011; Hilchey et al., Reference Hilchey, Klein, Satel and Wang2012). Although there is conflicting evidence (for review, see Mathot & Theeuwes, Reference Mathot and Theeuwes2011), some studies have found that the transfer of the full spatiotopic IOR takes several hundred milliseconds after the end of the saccade. Together, these studies suggest that spatiotopic maps might be updated on a relatively slower scale than the one typically studied in peri-saccadic paradigms (Zimmermann et al., Reference Zimmermann, Morrone and Burr2014a).
Third, the spatial contingency between adapter and probe during the experiment has varied across different designs. In some studies, spatiotopic trials (adapter and probe in the same spatial location) have been tested in separate blocks from nonspatiotopic trials, while in other studies the adapter and test were shown in the same location on only a small proportion of trials. The prediction in the latter case would differ for two alternative models of visual stability. If these nonretinotopic perceptual effects depend strongly on spatiotopy, then contingency should not matter. In contrast, other theories have emphasized the role of prediction and spatial updating in object-based remapping (Bompas & O'Regan, Reference Bompas and O’Regan2006; Melcher & Colby, Reference Melcher and Colby2008). For example, participants can learn color contingencies over a number of trials when the saccadic target changes color during the saccade, and this eventually leads to a color adaptation aftereffect (Bompas & O'Regan, Reference Bompas and O’Regan2006). Likewise, in a saccadic version of the McCollough effect (Ross & Ma-Wyatt, Reference Ross and Ma-Wyatt2004), making an eye movement to and from the colored adapters prolonged the duration of the aftereffect. These two studies suggest that contingency might be important for the way the brain generates visual stability, at least in the case of transient remapping: experimental paradigms, in which there is a complete lack of contingency between adapter and probe location, could actually train participants to dismiss a spatiotopic aftereffect related to remapping.
One final explanation for the discrepancy in results is that spatiotopic effects may interact with other phenomena active during peri-saccadic perception. Since the visual system is based on retinotopic coordinates, visual spatiotopy must be constructed, such as by an updating process and/or the alignment of retinal information with head, body, or allocentric reference frames in the brain. Many studies aim to understand this process by probing the transition periods, for example by flashing a stimulus immediately before or after a saccade, yielding phenomena like compression or displacement. In contrast, “spatiotopy” describes a more long-term process of visual stability in the ecological setting of everyday life. A stimulus presented shortly after saccade offset might interact with the complex set of processes involved in spatial updating and visual stability, and as described below these processes seem to differ in terms of their time course. Thus, for example, depending on the timing of the events in a trial, the tilt aftereffect shows a pattern of compression toward the saccade target in the peri-saccadic time period (Melcher, Reference Melcher2007; Zirnsak et al., Reference Zirnsak, Gerhards, Kiani, Lappe and Hamker2011) or a spatiotopic effect over a period of hundreds of milliseconds (Melcher, Reference Melcher2005; Zimmermann et al., Reference Zimmermann, Morrone, Fink and Burr2013a,Reference Zimmermann, Morrone and Burrb). Depending on the parameters of the experiment, either of these effects, or combination of both, could be reported.
The idea that basic visual features are encoded only in retinotopic coordinates has also been called into question based on studies using object motion, rather than self motion. As a first example, nonretinotopic encoding has been found using the Ternus–Pikler apparent motion display. When the two frames of the display are separated by a sufficiently long ISI (typically 100 ms), the three disks are seen as moving back and forth and features are combined across the object motion rather than based on retinal position (Otto et al., Reference Otto, Oğmen and Herzog2006, Reference Otto, Oğmen and Herzog2008, Reference Otto, Oğmen and Herzog2009; Boi et al., Reference Boi, Oğmen, Krummenacher, Otto and Herzog2009; Pooresmaeili et al., Reference Pooresmaeili, Cicchini, Morrone and Burr2012). Such studies have reported nonretinotopic effects for basic visual features, such as form and motion (Oğmen et al., Reference Oğmen, Otto and Herzog2006; Boi et al., Reference Boi, Oğmen, Krummenacher, Otto and Herzog2009). Further studies on iconic memory, visual priming, backward masking, and attentional cueing have used object motion displays to demonstrate object/frame-based coordinate systems in visual perception (Lin & He, Reference Lin and He2012; Lin, Reference Lin2013).
Second, nonretinotopic perception of feature information has also been found along the trajectory of motion (Otto et al., Reference Otto, Oğmen and Herzog2006; Nishida et al., Reference Nishida, Watanabe, Kuriki and Tokimoto2007). For example, chromatic color fusion rates at which color perception would normally be eliminated are possible if the color patches are perceived to be moving (Nishida et al., Reference Nishida, Watanabe, Kuriki and Tokimoto2007). Similar effects have been found for detection of a small spatial offset along a motion trajectory (Otto et al., Reference Otto, Oğmen and Herzog2006). Unlike the Ternus–Pikler display, which yields nonretinotopic feature processing only for longer ISI durations (>100 ms), the time frame for nonretinotopic color fusion (>20 Hz) is consistent with relatively early levels of visual processing.
A third line of nonretinotopic motion effects, perhaps also related to integration along a motion trajectory (Burr et al., Reference Burr, Ross and Morrone1986; Burr & Ross, Reference Burr and Ross1986), is the perception of intact objects when they are presented while moving through a small virtual slit (Zöllner, Reference Zöllner1862; Parks, Reference Parks1965; Ağaoğlu et al., Reference Ağaoğlu, Herzog and Oğmen2012). In the slit-viewing paradigm, also known as anorthoscopic perception, only a small fraction of the stimulus is seen at any moment in time, and always in the same retinotopic location, requiring temporal integration in order to perceive a coherent object. A careful analysis of the retinal image produced by these images confirms that object perception cannot be explained by eye movements or retinal smear of the stimulus but instead depends on integration of the information over time in a nonretinal reference frame (Rieger et al., Reference Rieger, Gruschow, Heinze and Fendrich2007).
In addition, perceptual learning has been demonstrated in nonretinotopic coordinates. Zhang and Li (Reference Zhang and Li2010) reported perceptual learning for motion direction discrimination transferred to a new stimulus shown in the same spatial, but different retinal, position. Nonretinotopic perceptual learning for orientation was reported using a novel paradigm in which the perceived location and orientation of a stimulus differed from the actual, retinal orientation and location (Otto et al., Reference Otto, Oğmen and Herzog2010). Critically, in both studies perceptual learning followed perceived, rather than retinal, coordinates.
In summary, a large number of studies have reported nonretinotopic perceptual effects in a wide array of behavioral paradigms that involve either self-motion or object-motion. It is useful to list these diverse paradigms as a first step in understanding this pattern of results and in eventually developing a comprehensive theory of when and how visual effects occur in nonretinotopic coordinates. More generally, the fact that the visual system encodes information, at least initially, in retinotopic coordinates raises the question of how the wide variety of nonretinotopic effects described above might arise at all. As described below, a number of neurophysiological and neuroimaging studies have discovered nonretinotopic mechanisms in the brain, including in visual areas, which might be useful for maintaining visual stability despite eye and body movements.
Evidence for nonretinotopic processing in nonhuman primates
Retinotopic mapping of receptive fields relies on experimental paradigms in which a stimulus is briefly presented while the observer maintains stable eye and head position. Typically, visual responses of neurons, including in the third and fourth visual complex (V3 and V4), have been studied during steady fixation or even often under anesthesia. In a small number of studies, however, the spatial and temporal attributes of receptive fields have been examined under more natural viewing conditions in which eye and/or head position changed. These experiments have revealed a number of nonretinotopically organized responses to visual stimuli.
Eye-position signals and gain fields
Gain changes in neural firing that are contingent on eye gaze position (“gain fields”: Andersen & Mountcastle, Reference Andersen and Mountcastle1983, Zipser & Andersen, Reference Zipser and Andersen1988) have been described in many areas within visual cortex. Studies on neurons in area V3A, for example, have shown that although they are retinotopic, both the visual response of the neuron, as well as the activity of the neuron in total darkness, are modulated by eye position. Similar findings were reported for area V6 (Galletti et al., Reference Galletti, Battaglini and Fattori1995). Gain fields have by now been reported in many areas of visual processing, including V1 (Trotter & Celebrini, Reference Trotter and Celebrini1999; Durand et al., Reference Durand, Trotter and Celebrini2010; Przybyszewski et al., Reference Przybyszewski, Kagan and Snodderly2014), V5/MT (Bremmer et al., Reference Bremmer, Ilg, Thiele, Distler and Hoffmann1997) and V4 (Bremmer, Reference Bremmer2000). Computational models have shown that gain fields provide sufficient information to encode visual information both retinotopically and craniotopically simultaneously within the same network of neurons (Zipser & Andersen, Reference Zipser and Andersen1988; Pouget et al., Reference Pouget, Fisher and Sejnowski1993).
Craniotopy and spatiotopy
In addition to retinal coordinates, the brain represents stimuli in other reference frames, both egocentric and allocentric. Multisensory perception of audio–visual stimuli, for example, requires information to be combined between retinotopic and craniotopic reference frames. The brain may also encode the location of objects in allocentric locations, since it is critical to know where items are located in space, not just with respect to specific receptors and effectors.
Consistent with this idea, it has been shown that the response of some neurons in V6 (Galletti et al., Reference Galletti, Battaglini and Fattori1993) and Ventral intraparietal area (VIP) (Duhamel et al., Reference Duhamel, Bremmer, Benhamed and Graf1997) depends on the location of the stimulus in space, not on the retina. This provided initial evidence for craniotopic (or spatiotopic) coordinate systems for visual neurons. Evidence for a more widespread presence for space-based, rather than retinal-based coordinate system for visual processing comes from studies of head tilt. For example, some V1 neurons have been shown to compensate for head tilt and ocular torsion, effectively responding in a spatiotopic manner (Daddaoua et al., Reference Daddaoua, Dicke and Thier2014). This complements previous studies showing that V1 neurons alter their orientation tuning to reflect head or body tilt, but only in awake (nonanesthetized) animals, consistent with an active mechanism (Denny & Adorjani, Reference Denny and Adorjani1972; Tomko et al., Reference Tomko, Barbaro and Ali1981; Durand et al., Reference Durand, Trotter and Celebrini2010).
Similar compensatory effects have been shown for motion during pursuit eye movement. Rather than signaling object image motion on the retina, a group of neurons in monkey Medial superior temporal area (MST) represent object motion in world-centered coordinates (Ilg et al., Reference Ilg, Schumann and Thier2004). Such data are consistent with the view that the coding of optic flow direction in monkey MST is specified in nonretinal coordinates and that the response of these neurons compensates for the effect of eye movements (Bradley, et al., Reference Bradley, Maxwell, Andersen, Banks and Shenoy1996).
Remapping of receptive fields
Perhaps the most interesting violation of retinotopy comes from studies showing the “re-mapping” of receptive fields around the time of saccadic eye movements. Duhamel et al. (Reference Duhamel, Colby and Goldberg1992) found that neurons respond shortly before a saccade to stimuli presented to positions that will fall in the receptive field after the saccade is made (thereby “anticipating” the effect of the saccade). Initially, these dramatic changes in the spatial and temporal profile of neurons were reported in parietal areas (Duhamel et al., Reference Duhamel, Colby and Goldberg1992), the frontal eye fields (Umeno & Goldberg, Reference Umeno and Goldberg1997), and the superior colliculus (Walker et al., Reference Walker, Fitzgibbon and Goldberg1995). More recently, however, neurons in areas of the third and fourth visual complexes have been shown to change their spatiotemporal selectivity around the time of saccades. Nakamura and Colby (Reference Nakamura and Colby2000, Reference Nakamura and Colby2002) reported robust re-mapping in areas V3A, V3 and V2 (for review, see Merriam & Colby, Reference Merriam and Colby2005). Some neurons shifted the location of sensitivity based on the upcoming saccade, with increasing response to the future RF location and decreasing response at the current RF. Other neurons seemed to expand the RF to respond to both locations up until the point of the saccade. In area V3A, for example, more than half the neurons responded to stimuli presented in the future RF even though it was extinguished prior to the saccade, and in many of these neurons the response was presaccadic (Nakamura & Colby, Reference Nakamura and Colby2002). It is also interesting to note, with regards to visual stability, that neurons in V3A respond based on actual stimulus motion more strongly than retinally matched self motion (Galletti et al., Reference Galletti, Battaglini and Fattori1990). Particular robust patterns of remapping were found in area V3A but also to a lesser extent in areas V3 and V2. The proportion of neurons showing remapping increased along the hierarchy of visual areas tested, while the latency of these remapping effects decreased for higher areas (Nakamura & Colby, Reference Nakamura and Colby2002). This pattern is consistent with a top-down signal, perhaps from LIP to V3A, which filters down to lower levels.
Neurons in area v4 have also been shown to dynamically change the spatial profile of their receptive fields around the time of saccades. Shortly before saccade initiation, in the presaccadic time period implicated in predictive remapping described above, RFs shift their spatial location toward the saccade target (Tolias et al., Reference Tolias, Moore, Smirnakis, Tehovnik, Siapas and Schiller2001). Neurons undergoing a spatial shift also tend to shift their responses in time, with an added latency delay of around 50 ms. A similar pattern of compression toward the saccadic target has also been reported in the frontal eye fields (Zirnsak et al., Reference Zirnsak, Steinmetz, Noudoost, Xu and Moore2014). It is interesting to note that the patterns of dynamic RF shifts in V4 and FEF differ from those reported in LIP (Duhamel et al., Reference Duhamel, Colby and Goldberg1992), the superior colliculus (Walker et al., Reference Walker, Fitzgibbon and Goldberg1995) and V2, V3 and V3A (Nakamura & Colby, Reference Nakamura and Colby2002), in which anticipatory shift of RFs take place toward the post-saccadic location of the receptive field. This different pattern of results of dynamic RFs, both within and across areas, suggests that there might be multiple types of remapping. In particular, the close link between the frontal eye fields and area V4 (compression toward the saccade target), on the one hand, and between LIP and V3A on the other (spatial shift), could support different mechanisms.
Remapping and other nonretinotopic effects have also been studied in motion processing areas MT and MST. Eye position modulations consistent with gain fields have been found in both MT and MST (Bremmer et al., Reference Bremmer, Ilg, Thiele, Distler and Hoffmann1997). The existence of remapping in motion areas may depend on the region under consideration. First, it was reported that neurons in area V5/MT did not show predictive remapping prior to saccades (Ong & Bisley, Reference Ong and Bisley2011). A more recent study looking at both MT and MST reported remapping of neurons in area MST but not area MT (Inaba & Kawano, Reference Inaba and Kawano2014, PNAS). One interpretation of the existing evidence, taking also into account the literature on optic flow described above, is that MT is mainly retinotopic while activity in MST is modulated by saccades. As described below, human neuroimaging studies have also reported changes in MST that reflect perceived motion (rather than retinal motion) and the influence of eye movements. This suggests that nonretinotopic responses in area MST may play an important role in visual stability across eye movements.
Human neuroimaging studies
Gain fields and spatiotopy
A number of studies have looked at nonretinotopic processing in humans using fMRI, in particular the properties of gain fields, of saccadic remapping and of spatiotopy. In terms of gain fields, a number of fMRI studies have reported changes in visual responses based on eye position (DeSouza et al., Reference DeSouza, Dukelow and Vilis2002; Merriam et al., Reference Merriam, Gardner, Movshon and Heeger2013; Strappini et al. Reference Strappini, Pitzalis, Snyder, Mcavoy, Sereno, Corbetta and Shulman2014). For example, activity in multiple visual areas, including V1 and V2, has been shown to be modulated by gaze position (d'Avossa et al., Reference d'Avossa, Tosetti, Crespi, Biagi, Burr and Morrone2007; Crespi et al., Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011; Strappini et al., Reference Strappini, Pitzalis, Snyder, Mcavoy, Sereno, Corbetta and Shulman2014). Merriam et al., Reference Merriam, Gardner, Movshon and Heeger2013 were able to decode eye position from the pattern of responses of voxels in visual areas including V1, V2, V3, V3A, and MT. In contrast, area V4 did not allow for eye position classification in that study.
Along the visual ventral stream, there is also evidence for spatiotopic adaptation of neural responses. McKyton and Zohary (Reference McKyton and Zohary2007) found adaptation of the BOLD response in the lateral occipital complex (LOC) when an object was repeated in the same spatial location even when the eye position was varied. This finding was taken as evidence for craniotopic or spatiotopic encoding by some neurons in LOC. Similarly, Golomb and colleagues found adaptation of neural responses in the perihippocampal place area when participants made scanning saccades across an image of a stable scene, consistent with spatiotopic encoding (Golomb et al., Reference Golomb, Albrecht, Park and Chun2011). No adaptation was found when the eyes were fixed but the scene was displaced under a matched condition, creating an equal retinotopic visual input to the active scanning condition but without the active visual signal. This finding suggests that information about eye movements might play a role in spatiotopic representations in ventral visual processing areas. Craniotopic maps have also been found in fMRI studies of human area VIP (Sereno & Huang, Reference Sereno and Huang2006). In this study, both visual images and facial air puffs were presented in order to map the response of neurons. Aligned visual and tactile maps were found in coordinates aligned with the face, rather than the retina.
As described above, one of the main tasks of the visual system is to distinguish between real motion (objects moving in the world) and retinal motion caused by self motion (eye, head movements, and locomotion). Initial evidence for craniotopic representation of motion information comes from a study of the influence of eye movement signals on the BOLD signal in areas MT and MST (Goossens et al., Reference Goossens, Dukelow, Menon, Vilis and van den Berg2006). It was reported that the response of area MT to optic flow stimuli was not highly influenced by making a pursuit eye movement (in contrast to studies in nonhuman primates: Bremmer et al, Reference Bremmer, Ilg, Thiele, Distler and Hoffmann1997). In contrast, and consistent with neurophysiological evidence, the response of human MST was modulated by pursuit eye movements. Using planar motion as opposed to heading-related 3D expansion stimuli, Fischer et al (Reference Fischer, Bulthoff, Logothetis and Bartels2012) showed that human V3A and V6 are heavily involved in the integration of planar motion signals with eye movements demonstrating a dominant contribution of these areas to head-centered motion perception and to perceptual stability during eye movements.
Neuroimaging evidence for nonretinotopic coding of the location of visual targets also comes from a memory-guided saccade task, in which participants made a range of different saccadic eye movements from different starting points to different ending points on the screen (Pertzov et al., Reference Pertzov, Avidan and Zohary2011). Although many voxels in the frontal eye fields and intraparietal sulcus responded based on saccade vector, the response of voxels in an area of the lateral intraparietal sulcus corresponded to the location of the target on the screen. Thus, these voxels responded based on saccade target location in spatiotopic or craniotopic coordinates.
D'Avossa and colleagues have reported fMRI evidence for spatiotopic responses of human area MT+ (2007). When subjects paid full attention to a peripheral motion stimulus by performing a direction discrimination task, the BOLD response of MT+ complex lost the selectivity for the position of the stimulus in the ispi- and contralateral visual space. Mapping the selectivity using four motion stimuli and three gaze directions, they observed a strong mapping in MT+ for the contralateral peri-personal space, independent of gaze direction, as shown in Fig. 3B. The spatial BOLD selectivity tuning of MT+ for left (red), central (black) and right fixation overlapped even though the retinal position differed by more than 14° for many stimuli.
Spatiotopicity of visual cortex by measuring BOLD responses to random-dot motion stimuli presented at various positions while subjects maintained fixation on one of three different gaze directions (±10° and zero corresponding to the dashed lines in A and B). Area MT (Pannel B), heavily involved in the perception of motion, showed a clear selectivity in external rather than retinal coordinates, whereas primary cortex V1 (Pannel A) was retinotopically selective. From responses like those of A and B, a spatiotopicity index for each voxel was computed (see text), which varies from −1 (blue in C and D) for perfect spatiotopicity to +1 (yellow) for perfect retinotopicity. C and D show flat maps of one example subject of the spatiotopicity index under two conditions of passive viewing (as in A and B), where subjects simply maintained fixation (but were free to attend to the motion stimuli), and a dual-task attentive condition, where they performed a demanding detection task at the fovea (Crespi et al., Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011). With passive fixation, most of the regions show a clearly spatiotopic response. But performing the attention-demanding foveal task causes these same voxels to become retinotopic. This effect, spatiotopicity with passive viewing becoming retinotopic when attention was confined to the fovea, occurred not only in area MT but also in areas MST, LO, and V6. However, primary and secondary cortex, V3, V3a and VP showed mainly retinotopic responses under both conditions. Reprinted with permission from Crespi et al. (Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011).
Similar to that used by Gardner et al. (Reference Gardner, Merriam, Movshon and Heeger2008), a spatiotopic index can be defined from the spatial selectivity curves (as those of Fig. 3B). The index is calculated by contrasting how well the data fit with a spatiotopic selective model against a retinotopic one. In detail, the index is given by:
$${\rm{SI}} = {{\left( {residS - residR} \right)} \mathord{\left/ {\vphantom {{\left( {residS - residR} \right)} {\left( {residS + residR} \right)}}} \right. \kern-\nulldelimiterspace} {\left( {residS + residR} \right)}}\quad ,$$where residS is the summed squared difference in BOLD response amplitude for the three fixation conditions for a spatiotopic alignment and residR for the retinotopic alignment. The index can vary between −1 for perfect spatiotopicity to +1 for perfect retinotopicity. Fig. 3C and 3D show how the index varied for the individual voxels of the occipital pole for 1 subject, when participants attended either to the motion stimulus or to fixation (Crespi et al., Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011). In contrast to the finding with attention focused on the motion target, when participants ignored the peripheral motion stimulus because they were performing a highly demanding attention task in the fovea, the responses became retinotopic. This effect, spatiotopicity with passive viewing becoming retinotopic when attention is confined to the fovea, was found in area MT and also MST, LO, and V6. In this study, primary and secondary cortex, V3, V3a and VP showed mainly retinotopic responses under both conditions (see Fig. 3). It is interesting to note that some voxels in the peripheral field of V1 showed a spatiotopic response, which is consistent with a study with nonhuman primates (Durand et al., Reference Durand, Trotter and Celebrini2010).
The finding that when attention is allocated to the fovea the selectivity of nearly all areas of the occipital lobe is retinotopic was first reported by Gardner et al. (Reference Gardner, Merriam, Movshon and Heeger2008). These authors interpreted the data as evidence against the existence of a spatiotopic coding in MT. However, the fact that the same voxels shifted their spatial coding, depending on whether the attentional demanding foveal task was performed, strongly suggests that attention may play an important role in building spatiotopic representation. Crespi et al., Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011 verified also with cross-validation techniques that the spatiotopic selectivity did not result from a high level of noise in all the 6 subjects tested, as suggested by Gardner et al. (Reference Gardner, Merriam, Movshon and Heeger2008). However, Crespi et al. (Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011) did not directly manipulate the attentional load nor the type of attention (to the motion or to the object or to the position) and so further experiments are required to fully understand the phenomenon and the attentional modulation.
It is important to stress that the motion stimuli used in the spatiotopicity of motion integration (Melcher & Morrone, Reference Melcher and Morrone2003) and in the Functional magnetic resonance imaging (fMRI) studies of d'Avossa et al. (Reference d'Avossa, Tosetti, Crespi, Biagi, Burr and Morrone2007) and Crespi et al., (Reference Crespi, Biagi, d'Avossa, Burr, Tosetti and Morrone2011) are long lasting, from 3 to 16 s. Given that spatiotopicity takes time to construct, it is highly possible that a sequence of abrupt onset–offset stimuli would not produce similar spatiotopic processing. Interestingly, Golomb & Kanwisher (Reference Golomb and Kanwisher2012) performed an fMRI experiment presenting stimuli in three locations at two different fixations, and using both multivariate pattern analysis and the spatiotopicity index of Gardner et al. (Reference Gardner, Merriam, Movshon and Heeger2008) failed to reveal any spatiotopic selectivity in any part of the visual brain. They used categorical images of faces, bodies, and objects, following McKyton and Zohary (Reference McKyton and Zohary2007). In principle, a powerful technique like multivoxel correlation analysis should have highlighted a spatiotopic code at least in LO, while the analysis based on the spatiotopicity index could have missed it, given that the index computed with 3 position and 2 fixation biases toward retinotopy. Interestingly, in McKyton and Zohary's (Reference McKyton and Zohary2007) experiment, the stimulus was always present for the 12 s in the spatiotopic or retinotopic position, while in Golomb and Kanwisher's (Reference Golomb and Kanwisher2012) the presentations were very brief and interleaved with 500 ms of blank. The strong retinotopic responses to these abrupt onsets might have masked the spatiotopic response that should be in any case very weak in those conditions given the limited time to be computed. Clearly, for categorical images, more experiments are required to resolve the conflicting results in literature.
As described above, neurophysiological investigations of remapping have also shown that it is limited to salient, attended items (for review, see: Melcher & Colby, Reference Melcher and Colby2008; Wurtz et al., Reference Wurtz, Joiner and Berman2011). Thus, attention may play a general role in nonretinotopic visual processing, as a mechanism to ensure that feature binding remains intact even without retinotopic matching of features.
The selectivity for spatiotopicity along the hierarchy of visual areas parallels very well with the selectivity of the cortical response to the perceived (rather than the retinal) position of a stimulus (Fisher et al., Reference Fischer, Spotswood and Whitney2011). Using a multivariate pattern analysis to track the coupling of the BOLD response with incremental changes in physical and perceived position during fixation, these authors found that activity in higher level areas—V3A, lateral occipital cortex, middle temporal complex for the dorsal pathways, and V4, LO, posterior fusiform gyrus, fusiform face area, parahippocampal place area for the ventral pathways—more precisely reflected the reported positions than the physical positions of the stimuli. Apparent position of a brief stimulus is also altered by surrounding motion. Interestingly, both V3A and MT spatial coding reflect these shifts in perceived position (Maus et al., Reference Maus, Fischer and Whitney2013), reinforcing the idea that spatial coding in these higher motion areas is flexible and reflects perceived location rather than only retinal coordinates.
All of these results indicate that the localization of external visual stimuli is an active phenomenon achieved by adapting to the various frames of reference. This also suggests that to conceptualize the problem in terms of retinotopy can be overly reductive. The brain integrates different types of sensory and cognitive signals to compute localization already at an early stage of analysis, likely as early as area V3.
Evidence for spatial remapping in humans
Several studies have adapted paradigms previously used in neurophysiological experiments to examine the neural response to stimuli presented around the time of saccades. In one set of studies using fMRI, a stimulus was flashed to the right or left of fixation (Medendorp et al., Reference Medendorp, Goltz, Vilis and Crawford2003; Merriam et al., Reference Merriam, Genovese and Colby2003, Reference Merriam, Genovese and Colby2007). In the no-saccade conditions, the flash evoked a contralateral response in the brain as expected. On saccade trials, participants made a horizontal saccade in such a way that the stimulus would be projected to the other visual hemifield in the subsequent fixation, but the stimulus was extinguished prior to saccade onset. Under these conditions, an increase in the BOLD signal was also found in the ipsilateral hemifield (the future, or remapped, location of the stimulus). This remapping of activity for the extinguished stimulus was reported in parietal cortex (Medendorp et al., Reference Medendorp, Goltz, Vilis and Crawford2003; Merriam et al., Reference Merriam, Genovese and Colby2003) and also in visual areas (Merriam et al., Reference Merriam, Genovese and Colby2007). In the latter study, the largest responses on remapping trials were found in areas V4 and V3A. Progressively smaller, but still significant, responses were found in V3, V2, and V1. Overall, these findings are consistent with neurophysiological data, in terms of largest remapping signals in V4 and V3A, but it is interesting to note that remapping responses in V1 found with fMRI exceeded those found in nonhuman primates.
A further methodology to measure spatial remapping is to measure adaptation in the BOLD signal when the same stimulus is repeated in the same screen coordinates across a saccade (McKyton & Zohary, Reference McKyton and Zohary2007; Golomb et al., Reference Golomb, Albrecht, Park and Chun2011). Preliminary evidence for such adaptation in visual cortex in humans comes from a recent fMRI study by Zimmermann and colleagues (Zimmermann et al., Reference Zimmermann, Morrone and Burr2014a,Reference Zimmermann, Weidner and Finkb). Behavioral and neural adaptations were observed under both the spatiotopic and retinotopic conditions. Significant neural adaptation effects were found in early visual areas V1 and V2 as well as in higher-tier visual areas including V3 and V4. No significant adaptation occurred in the control condition, thus ruling out spatially unspecific adaptation. Overall these data indicate that spatiotopic adaptation requires an active remapping of adapter activation to the new retinal position that coincides with the physical adapter location after the saccade.
Additional support for presaccadic remapping in visual areas comes from Electroencephalography (EEG) studies using a similar paradigm (Bellebaum & Daum, Reference Bellebaum and Daum2006; Parks & Corballis, Reference Parks and Corballis2008, Reference Parks and Corballis2010). As in the fMRI studies, an increase in ipsilateral activity in parietal and occipital sensors was found. Given the better temporal resolution of EEG compared to fMRI, it was possible to show that these modulations in ipsilateral responses began prior to saccadic onset. Together, results from fMRI and EEG are consistent with saccadic remapping in visual areas in the human brain, possibly as a result of top-down signals from LIP and the FEF.
Attention effects in different coordinate systems
Attention is tied to a number of nonretinotopic effects that have been reported in the visual system. Remapping is thought to be limited to items with high attention priority (for review, see: Melcher & Colby, Reference Melcher and Colby2008; Wurtz et al., Reference Wurtz, Joiner and Berman2011). This makes sense because items that are not attended are poorly represented in the brain and thus would not need to be painstakingly updated across each saccade. Given the limit in the number of items receiving full attention, the remaining visual field is likely summarized in “gist” or “ensemble” representations (for review, see: Melcher & Colby, Reference Melcher and Colby2008; Alvarez, Reference Alvarez2011), along with a general allocentric/spatiotopic spatial map (for review, see: Burr & Morrone, Reference Burr and Morrone2011; Jeffrey et al., Reference Jeffery, Jovalekic, Verriotis and Hayman2013). In the case of gist and ensemble representations, any single saccade would be unlikely to dramatically change the visual scene. Indeed, ensemble representations appear to persist over an extended period of time and to involve both retinotopic and nonretinotopic coordinate frames (Corbett & Melcher, Reference Corbett and Melcher2014; Fischer & Whitney, Reference Fischer and Whitney2014).
As described above, there is evidence that IOR occurs in spatiotopic coordinates when there is an intervening saccade (Sapir et al., Reference Sapir, Hayes, Henik, Danzinger and Rafal2004; Mathot & Theeuwes, Reference Mathot and Theeuwes2011; Pertzov et al., Reference Pertzov, Avidan and Zohary2011; Hilchey et al., Reference Hilchey, Klein, Satel and Wang2012). By presenting the probe at varying times after saccade offset, it is possible to map the time course of the switch from the former retinal position (which no longer corresponds to the spatial location of the target) to the new spatiotopic location. More generally, however, any shift of attention in the brain, whether driven by endogenous cues, exogenous cues, or saccades, can take hundreds of milliseconds. For example, studies on attention shifts show that it takes 200–500 ms before the new location reaches its highest level of attention benefit and the old location completely loses any attention effect (Duncan et al., Reference Duncan, Ward and Shapiro1994; Moore et al., Reference Moore, Egeth, Berglan and Luck1996; Ward et al., Reference Ward, Duncan and Shapiro1997; Theeuwes et al., Reference Theeuwes, Godjin and Pratt2004). Saccades also require the disengagement of attention from one set of neurons and transfer to another set of neurons that will respond to the new spatial location of the attended target. Thus, it is not particularly surprising that there would be lingering effects of attention at old locations and a delay in updating the attention focus for a period of a few hundred milliseconds, just as in any other study of attention shifts (Golomb et al., Reference Golomb, Chun and Mazer2008; Mathot & Theeuwes, Reference Mathot and Theeuwes2011). What is more surprising about those studies is that attention effects do emerge in spatiotopic coordinates over time, which must require some sort of active updating process.
An additional attention effect which influences perception and retinotopy is the presaccadic shift of attention to the target. Saccades are yoked to these presaccadic attention shifts, such that performance at other locations suffers while the saccadic target location is improved (Kowler et al., Reference Kowler, Anderson, Dosher and Blaser1995; Deubel & Schneider, Reference Deubel and Schneider1996). More recently, as described above, this presaccadic shift has been visualized as a compression of receptive fields toward the saccade target. In practice, however, it is difficult to distinguish a pure attention shift (which improves performance) from remapping based on classic behavioral methods (percentage correct or reaction time measures) or evoked responses. Better or worse performance, or higher or lower evoked responses, could be a result of the attention effect or the remapping effect, unless these two are careful dissociated by the experimental design. Two examples are peri-saccadic unmasking (De Pisapia et al., Reference De Pisapia, Kaunitz and Melcher2010) and peri-saccadic un-crowding (Harrison et al., Reference Harrison, Mattingley and Remington2013). In both studies, targets were backward masked in time, raising the question of whether the improvement in performance (measured only in terms of percentage correct) can be ascribed to a general effect of the attention shift or instead involves remapping (van Koningsbruggen & Buonocore, Reference van Koningsbruggen and Buonocore2013).
A final issue is the debate regarding whether attention is “retinotopic” or “spatiotopic” in nature. We would argue that this is an ill-posed question, since “attention” can be operationalized in terms of the selective modulation of sensory processing as well as the selection among competing memory representations, tasks, etc. In the case of vision, spatial attention acts in retinotopic coordinates by modulating sensory processing in visual areas. However, attention to sound (the location, pitch or loudness of an auditory tone) or touch (the location or type of tactile stimulation) would not be retinotopic. Selecting a memory or switching tasks would also likely not involve a single retinotopic coordinate frame. Thus, attention acts on processing in various reference frames. The critical point for perceptual stability is that the brain is able to keep track of these attended items even when the object or the body (eyes, head, hands, etc.) moves. It is likely this need to deal with object-motion and self-motion that necessitates the mechanisms that underlie nonretinotopic effects.
Emerging ideas about spatiotopicity and its potential mechanisms
The problem of creating a stable percept of the world is incredibly complex, involving neural representations for navigation through and rapid interaction with our environment, as well as providing the rich perceptual experience of a high-resolution, stable world. Given the evidence that space is encoded in multiple representations in the brain (Colby & Goldberg, Reference Colby and Goldberg1999), it seems highly likely that multiple mechanisms for spatial stability coexist. Based on the current evidence for remapping and spatiotopicity in monkey and human brains, we suggest that both mechanisms operate together to mediate perceptual stability. Remapping is a rapid-acting, high-resolution system of transient spatiotopy, based largely on retinotopic representations that dynamically adapt on each saccade, bridging the disruption of the saccade to provide trans-saccadic perceptual continuity. This necessarily would require the work of small receptive fields and should act already at a relatively early stage of visual processing. In this case, remapping is a transient version of nonretinotopic processing based on a spatial and temporal transition between different retinotopic maps.
In contrast, spatiotopic selectivity should be a low-resolution system of maps that is invariant with eye and body position and operates over a longer time frame. The low resolution of this map is consistent with the larger RF size of higher areas where spatiotopicity has been observed, such as LO and MT. This coarse representation of the world in real-world coordinates is also in line with the large errors in localization that are so common in our perception. Converging evidence suggests that spatiotopicity is also tightly linked with perceptual memory and develops over time in order to support spatial cognition over a longer time frame (for review Zimmermann et al., Reference Zimmermann, Morrone and Burr2014a). Moreover, spatiotopic representation is strongly modulated by high-level cognitive factors such as attention. Consistent with ideas suggesting that spatial perception and action is guided by an attentional priority, or saliency, map (Gottlieb, Reference Gottlieb2007; Melcher & Piazza, Reference Melcher and Piazza2011; Knops et al., Reference Knops, Piazza, Sengupta, Eger and Melcher2014), it would be efficient to maintain only behaviorally relevant items in a spatiotopic map.
In future work, it will be important to look at the role of time in neural signatures of nonretinotopic processing and to include more studies using stable visual stimuli (objects) instead of unpredictable, flashed visual probes. As described above, current behavioral evidence suggests that spatiotopic representations develop and are updated over time, and that nonretinotopic processing may be linked to attention and to stable objects. In contrast, most stimuli used in neurophysiological and neuroimaging studies have been briefly flashed stimuli. These experiments fail to capture the time course of stimuli in the natural world, in which self-movement (in particular saccadic eye movements) displaces the location of already present stimuli across the retina rather than stimuli appearing at random locations and timings on a stationary retina. In sum, future work on the role of nonretinotopic processing in visual stability may need to include experimental contexts in which the visual world is actually stable.
Acknowledgment
MCM was supported by the European Research Council (FP7; Advanced Grant - Early Sensory Cortex Plasticity and Adaptability – 338866) and DM was supported by the European Research Council (FP7; Starting Grant - Construction of Perceptual Space-Time, grant agreement 313658).
