R3.1. Motor compensation and accounts of flash-lag

There are many types of prediction – motor, visual, and cognitive – and Kerzel & Müsseler underscore this generality. However, phenomena of such generality would require mechanisms of comparable diversity. Consequently, predictive mechanisms may be widespread within the nervous system. This also stands to reason, as the brain is a massively interacting computational device. Any significant neural function in one part of the brain (e.g., motor) can influence another part of the brain, such that both parts will show that function. But Kerzel & Müsseler suggest that visual mechanisms do not contribute to prediction. This is unparsimonious, as more that 50% of the cortex serves a visual function.

Sheth & Wu suggest that, “The agent can often over-compensate for the movements of the target, thus arriving at a common intersection point some time before the target,” in claiming that motor prediction is more flexible in compensating for delays in some situations, as it can tolerate slop. A clear example where compensation for delays may not be necessary is where a stationary animal hunting for fish is positioned in a narrow, fast-flowing stream. However, some tasks, such as a frog striking a flying insect with its tongue, would be impossible without extrapolation in visual neurons (possibly in the tectum). Furthermore, in the given fishing example, and myriad other similar examples, even skill (which Cutting favors) and motor compensation (which Sheth & Wu favor) are not required. Just as this example does not diminish the importance of skill and motor compensation generally, examples in which visual compensation may seem unnecessary do not diminish the importance of visual compensation generally. The animal needs to localize not only objects, but also its end-effectors. Prediction in the motor domain but not in the visual domain would result in a spatial mismatch between visual objects and end-effectors, which would in turn interfere with the formation of a unified spatial map. A mechanism for “motor-sensory unification” is considered in section R6.5.

Purushothaman, Bedell, Öğmen, & Patel [Purushothaman et al.] argue the point that compensation for delays is carried out by motor processes. It must be noted, however, that there is debate neither in the literature nor in the target article challenging the extremely important role of motor processes. The examples these commentators give are useful; however, these do not rule out the possibility of predictive visual processes. Although Purushothaman et al. are correct in saying that most of the evidence supporting prediction involves behavior (e.g., saccades), they fail to note that “purely” visual processes are not as divorced from motor processes as was once believed (Wilson & Knoblich Reference Wilson and Knoblich2005).

Smeets & Brenner misattribute the following claim to the target article: “In his target article, Nijhawan proposes that early visual processing provides the prediction that is needed to deal with sensory-motor delays when we interact with moving objects, rather than such prediction arising from complex motor strategies as is generally assumed.” Visual prediction does not replace motor strategies, but rather, suggests that prediction in vision either complements or reflects prediction in other (non-visual) processes. In addition, VP could provide crucial data not just for interactions but also for decision processes such as when to act, and whether to act or withhold action.

Sheth & Wu suggest that VP makes only a minor contribution to the flash-lag effect, invoking multiple other factors. In some cases, large magnitude effects, for example, some geometrical illusions, have been difficult to account for with a single mechanism, so multiple mechanisms have been invoked (Coren & Girgus Reference Coren and Girgus1973). However, there are still unresolved issues of basic causes versus modulatory influences for the flash-lag effect.

Baldo & Klein have championed an attentional framework for understanding the flash-lag effect. There is no doubt that attention has a role to play in the flash-lag effect, however, what this role is remains unclear, as there are discordant findings (Baldo & Klein Reference Baldo and Klein1995; Khurana et al. Reference Khurana, Watanabe and Nijhawan2000; Khurana & Nijhawan Reference Khurana and Nijhawan1995). Another difficulty for the attentional framework is that there are many forward-shift effects that do not involve flashes (e.g., see Maus & Nijhawan Reference Maus and Nijhawan2006), so movement of attention is not present, while forward-shift effects are.

R3.2. Additional delays in the localization of moving objects would be dangerous

Smeets & Brenner claim that the visual system determines the precise location of a moving object only when there is a need to know the object's location, not throughout the object's motion trajectory. They suggest that the flash-lag effect occurs because the observer needs to determine the position of the object after the flash is presented.

I claim that the visual system acquires and has continuous position information of a smoothly moving object. This is because precise position of a moving object could be required to guide action at an unpredictable instant. Localization is a fundamental task, and the mechanisms serving localization are frequently tested (and selected) in trying situations, such as predator–prey interactions. On account of survival pressure, animal nervous systems cannot tolerate any added delays in localization of moving objects. Continuous location information concerning a moving object has a survival advantage, as the relevance of a moving object can change drastically from one moment to the next.

Imagine a monkey being hunted by a tiger, which is slowly approaching from the right. The monkey is not unduly concerned, as there is a sure getaway – there is a large tree near the monkey to the left that it can quickly climb. Now imagine that there is second tiger hiding behind the tree, whose presence is given away by the sound of a twig breaking under the tiger's paw. Under this scenario, and innumerable similar scenarios in nature, the animal that already has location information regarding an important moving object (in this example, the approaching tiger from the right) will be able to plan its next escape strategy more quickly, and thus have a survival advantage over an animal that must first establish the location of the moving object following an unpredictable event (in this example, the sound of the twig).

R3.3. The flash_φ/flash_ψ muddle

The non-controversial aspect of the various accounts of the flash-lag effect is the assumption of a significant delay between the presentation of a flash and its perception. However, keeping the two events, flash presentation and perception, separate is as difficult as it is important to be explicit as to which of the two meanings one intends when using the term flash. In order to be more precise, I will denote the event physical-flash by flash_φ and the event psychophysical-flash by flash_ψ. Note that the event flash_ψ is short for the perception of the flash.

Now let us turn to the moving object. Visual input delay (Δt _i) should cause an observer to perceive a continuously moving object in a position that it occupied Δt _i seconds ago. The initial impetus for the suggestion that this statement is incorrect came from the flash-lag effect (Nijhawan Reference Nijhawan1994), and motion extrapolation was suggested as a process that undoes this lag. Happily, Eagleman suggests that postdiction, which until now has been considered to be the antithesis of prediction, can also achieve this at least partially.

The initial basis for the postdiction account was the observation that the flash-terminated condition does not lead to the flash-lag effect (Eagleman & Sejnowski Reference Eagleman and Sejnowski2000). Postdiction suggested that the visual system integrates information after an event, and this integrated information can impact perceptual decisions about an event in the immediate past. A semantic analogy would be a word appearing later in a sentence giving added precision to (or disambiguating) the meaning of a word appearing earlier in the sentence. Eagleman suggests that continuous input from future events could update past delayed information to align the perceived position of a moving object with its actual position. But what happens when a flash is presented? On the current rendition of postdiction, when the moving object is physically aligned with flash_ψ, it is also perceptually aligned with it (as the delay between the perceived and physical position of the moving object has been removed by the postdictive process). Thus, when event flash_ψ occurs, the perceived position of the moving object is ahead, which explains the flash-lag effect. However, on this view, a flash-lag effect should also occur in the flash-terminated condition. The lack of effect in the flash-terminated condition now poses a problem for postdiction, even though it was the initial impetus for it.

Now let us consider the flash-initiated condition. Although it is correct that motion information after event flash_ψ (which is what is commonly meant by the term “flash”) drives the flash-lag effect, it is equally correct to say that motion input before event flash_ψ drives the flash-lag effect. The suggestion by Smeets & Brenner that the dependence of the flash-lag effect on what happens after the “flash” (more precisely, flash_ψ) opposes VP, is incorrect. By definition, in the flash-lag effect, the observers report that flash_ψ lags behind the moving object, so flash_ψ is the all-important event. Therefore, this event should play the central role in all analyses. According to the analysis presented in section 5.2.1 of the target article (also see Figure 7 there), more than 98% of the time period immediately before flash_ψ, during which the moving object is in view, is available for motion extrapolation. So, it is the past, not the future, motion that contributes to the flash-lag effect.

Eagleman believes apparent motion to be a quintessential postdictive phenomenon. In apparent motion, two dots, dot 1 and dot 2, are flashed in two positions in quick succession. The perception is that dot 1 appears to move to dot 2. Movement of the dot could have started only after dot 2 appears, and yet the movement of the dot implies that it occupies positions between dot 1 and dot 2, which are past positions. This phenomenon has puzzled many researchers, and Eagleman has identified a possible postdictive account of this. I believe this phenomenon seems puzzling due to acquiescence. Influential scientists have continually reminded us that we have no direct immediate access to the world (Neisser Reference Neisser1976) and that visual perception is a generative process that “creates” the visual image (Kandel & Wurtz Reference Kandel, Wurtz, Kandel, Schwartz and Jessell2000). In the absence of such reminders, we acquiesce into believing that our visual percepts directly reflect the state of the world.

There is a temporal version of this acquiescence, which results in the flash_φ/flash_ψ muddle and engenders a postdictive account of apparent motion. It is well known that motion can be perceived even when there is no object appearing to move. Sitting in a quiet room, we can sometimes have the experience that something moved, without perceiving what moved. In the far periphery of our retina, a stimulus is perceived only when it is moving; as soon as movement stops, the object vanishes. This is what happens in apparent motion: say event flash1_φ occurs at time 1 and flash2_φ quickly follows at time 2. This leads to neural signaling of motion. These motion signals occur before event flash2_ψ (because flash2_ψ is subject to conduction delays). Thus, at the time of flash2_ψ, the motion computation has already occurred (on the basis of flash2_φ), so the motion percept is interjected before the percept of the second flash. The seemingly postdictive nature of apparent motion comes about because we forget that flash2_φ and flash2_ψ are distinct events.

In conclusion, postdiction does not cause the flash-lag effect, but it does contribute to its elimination it in the flash-terminated condition. Backward masking in which a later stimulus renders an earlier stimulus invisible is another excellent example of a postdictive phenomenon. By letting later information have an impact, postdiction has the advantage of obtaining more reliable data about the world, and disambiguating information when the data are noisy.

R3.4. Going nowhere by misinterpreting and mislabeling visual prediction

Öğmen, Patel, Purushothaman, & Bedell [Öğmen et al.] and Purushothaman et al. appear to have misinterpreted the basics of VP. They even mislabel it as “perceptual compensation model” (PCM), and then proceed with criticisms that are directed at PCM and not VP. At first, it might seem that the authors may have simply switched the word “perceptual” for “visual,” as one may interchangeably use the terms in describing the perceptual system or the visual system. This would not be a serious problem. However, there is a deeper issue, as revealed in several statements by Öğmen et al. and Purushothaman et al. Consider the following statement by Purushothaman et al.: “Perceptual compensation in early visual areas, as suggested by the target article, need not occur.”

In the target article I suggested that perception is the outcome of visually sensitive neurons that are capable of compensating for visual delays, or that compensatory motor processes can influence perception (e.g., via feedback from motor areas to visual areas). The percepts reveal this compensation (e.g., in the flash-lag effect) because of compensation mechanisms in the feedforward vision pathway and/or motor pathways. Thus, perception is not a process that produces compensation, as the authors seem to suggest. Rather perception is the outcome of compensation mechanisms.

Öğmen et al.'s misinterpretation of VP is underscored when they draw a contrast between perceptual compensation (their term) and sensory compensation. They say: “That sensory processes may be predictive is neither new nor controversial… . What is controversial is the specific hypothesis proposed by Nijhawan, namely, compensation of neural delays at the perceptual level.” It is, however, highly unlikely that sensory processes can be predictive while the perceptions that these sensory processes contribute to are not.

Contrary to what Purushothaman et al. report, not only does pursuit eye-movement yield a flash-lag, but the magnitude of the effect with pursuit is about the same as in the standard object motion case (Nijhawan Reference Nijhawan2001; also see van de Grind's commentary). Purushothaman et al.'s suggestion that other motion-bias phenomena (De Valois & De Valois Reference De Valois and De Valois1991; Ramachandran & Anstis Reference Ramachandran and Anstis1990) do not support the notion of compensation for delays, is not widely shared (see, e.g., commentary by Whitney). Finally, Öğmen et al.'s favored explanation of our fading motion data in terms of lowered detection threshold resulting from integration along the motion path, is incorrect; we explicitly tested this hypothesis by doubling the trajectory length of the motion stimulus and found the detection threshold to not change significantly (Maus & Nijhawan Reference Maus and Nijhawan2006, p. 4379).

Öğmen et al. claim that the target article is selective, as it does not address the luminance-dependent modulation of the flash-lag effect. This is surprising, as section 6.1. of the target article is devoted to this very issue. I would like to add that many everyday objects consist of parts of different luminance (e.g., a fair-skinned person with black hair), and movement of such objects does not lead to spatial distortions. This supports the view that the visual system takes differential latency (due to variations in luminance) into account. This brings me to a related point concerning the “flash-lead” effect (Purushothaman et al. Reference Purushothaman, Patel, Bedell and Öğmen1998), which the commentators mention. The authors have claimed that a flash-lead effect occurs when the display combines a moving object of low luminance contrast with a bright flash. Let me say that in all my experience of working with the flash-lag effect since 1990, I have never observed a flash-lead effect based on luminance differences. This is also true of displays that combine a moving object that is close to absolute detection threshold and a stroboscopically generated flash that is a million times brighter! This has also been noted by others (see, e.g., Brenner & Smeets Reference Brenner and Smeets2000, and Smeets & Brenner's commentary).

R3.5. Neural response to the instantaneous value of continuous input

In the interaction between a monkey and a tiger that Krekelberg considers, the smooth changes in direction that the tiger produces are more to test the monkey's endurance than to challenge its visual system. What is demanding for the visual system in predator–prey interactions is the sudden changes in movement direction that the animals often produce. Such energetically costly movements are selected precisely because smooth changes in movement are predictable.

There have been several studies on motion per se, showing predictive responses of neurons (e.g., Fukushima et al. Reference Fukushima, Yamanobe, Shinmei and Fukushima2002). Krekelberg's interpretation that a change in the response maximum of a middle temporal area (MT) neuron to different motions shows that this neuron is unable to compensate for the delay, is incorrect. The possibility of relating the neural response of a cell to the “wrong” instantaneous value of a continuously changing input was discussed in the target article for orientation tuning in primary visual cortex (V1) (see sect. 1.3). There is a relatively straightforward interpretation of the interesting results shown by Krekelberg. The so-called directional label of a MT neuron depends not just on the intrinsic properties of the neuron, but also on the context of the stimulus. In the present case, the directional tuning curve of an MT neuron depends on the history of the stimulus. In general, if the movement direction of the stimulus changes continuously, it will produce a different tuning curve than if the stimulus moves in a fixed direction within the MT cell's receptive field (in the vast majority of studies, fixed direction stimulus is used). Furthermore, the tuning for clockwise motion versus counterclockwise motion would be different, as the history of the stimuli is different for the two motions. Continuous input to a cell leads to adaptation in the cell's response, and clockwise (CW) and counterclockwise (CCW) motions will cause differential adaptations. In addition, a single neuron does not receive input only from the retina, but also from other neighboring cells, which are differently tuned. These factors, and not differential delay in the cell's peak response to CW and CCW motions, are the likely cause of the observed difference in the peak response of the cell described in the commentary.

R3.6. Prediction and violation of prediction in the flash-lag phenomenon

Baldo & Klein suggest that the original motion extrapolation account (Nijhawan Reference Nijhawan1994) nicely explains the flash-lag effect in its purest form. They state, however, that:

Baldo & Klein are correct in saying that the target article focuses more on new physiological mechanisms, but neither have these mechanisms been mentioned here for the first time, nor are they introduced to explain discordant findings.

Take the role of transients in the localization of moving objects, which was first introduced by Nijhawan (Reference Nijhawan2002). In nature, predictable events and events that violate predictions go hand in hand, so one goal of the target article was to look at both sides of the coin. Prey animals often produce abrupt trajectory changes in an attempt to throw off a following predator. There is recent neurophysiological evidence that the retina not only predicts moving objects but also strongly signals violations of prediction (Holy Reference Holy2007; Schwartz et al. Reference Schwartz, Taylor, Fisher, Harris and Berry2007b) resulting from sudden changes in the trajectory of moving objects. It is natural to think of an abruptly disappearing moving object in the flash-terminated condition, or sudden change in object trajectory in the motion-reversal condition, as violating prediction. I suggest that the strong transient signals generated by violations of prediction interact postdictively (Eagleman & Sejnowski Reference Eagleman and Sejnowski2007) with predictive signals and overwrite them. We tested this claim by making the stimulus disappear without this violation and found the predictive-overshoot (Maus & Nijhawan Reference Maus and Nijhawan2006) that is typically missing in the flash-terminated condition. Furthermore, when the strength of the transients is manipulated, transients of intermediate levels result in intermediate forward displacements (Maus & Nijhawan, in press).

Howe, Horowitz, & Wolfe [Howe et al.] tested our account by doing the opposite experiment: they attempted to eliminate the flash-lag effect by introducing a transient. In their experiment, as an object moved on a gray background, it abruptly reversed contrast at the time of the flash from black to white. The observers reported a flash-lag effect. The commentators claim that this contradicts the proposal by Nijhawan (Reference Nijhawan2002) and Maus and Nijhawan (Reference Maus and Nijhawan2006) that the transient in the flash-terminated condition eliminates the predictive-overshoot. Before analyzing Howe et al.'s interesting experiment, it is important to note that there is a transient at the beginning of motion in the flash-initiated condition, which fails to eliminate the flash-lag effect (Khurana & Nijhawan Reference Khurana and Nijhawan1995). So, clearly, the timing of the transient is crucial.

One can think of the contrast reversal experiment as consisting of two phases. The first phase, with a black moving object, comprises the flash-terminated condition, and the second phase, with a white moving object, comprises the flash-initiated condition. We know that the latter condition does and the former condition does not produce a flash-lag. For a black-to-white transition, the important question to ask, however, is: Did the flash appear to lag the black or the white moving object? (It is not sufficient simply to report a flash-lag effect here). If the white object appeared ahead of the flash (which would be my guess), then this is not surprising, as this is just the flash-initiated condition producing the effect. In the biased competition model presented in the target article (see sect. 5.2.2.) the visual system gives higher weighting to newer retinal input. In the case of the flash-initiated condition, the initial transient resulting from motion onset does not fix the position of the moving object because there is newer (motion) input after the transient.

It would be of great interest if the black object appeared ahead of the flash. This would be the flash-terminated condition producing a predictive-overshoot (which is normally not seen). Such a finding would mean that the motion input resulting from the white object following contrast reversal weakens the black-to-white transient. A weakened transient, in turn, is less able to suppress the prediction-based representation of the black object. In conclusion, a transient will fix the location of a moving object only if (1) it comes at the end of an event (as a final event), so that it has the opportunity to compete strongly with, and mask, the previously setup prediction-based representation, and (2) no new input is there to suppress the transient.

R3.7. TTC models and VP

Benguigui, Baurès, & Le Runigo [Benguigui et al.] suggest that the visual system is not an accurate predictor of moving objects if the last part of their trajectory is occluded for more than 200 msec, contrary to the claim made in the target article, which suggests that the time period is much longer. However, the 200 msec limit does not significantly change my conclusion. Time-to-collision (TTC) theorists suggest that the nervous system uses information contained in the changing retinal image to compute TTC. In the target article, I claim (and make a case) that a corollary of the TTC approach is that visual mechanisms compensate for the delays that would be incurred in the transmission of the changing retinal image to the central nervous system (CNS). This claim will only be affected if accuracy in prediction is compromised for an occlusion duration that is close to the duration of visual processing delay. Since, according to Benguigui et al. prediction ability is compromised for occlusion durations of 200 msec and beyond, which is longer than visual processing latency, my claim is unaffected. So the restated claim would read: Because TTC can be predicted accurately 200 msec into the future on the basis of retinal information, the position of the moving object can certainly be predicted ~80 msec into the future, as required by the motion extrapolation account of the flash-lag effect.

R4.1. Are tolerance for delays and skill enough to account for human performance?

McFarland points out that the absolute reference location of a moving object is unavailable. Adding to the suggestions of van de Grind, I argue that the need for a reference for locating moving objects may not arise, as motion is absolute (see sect. R2). Alternatively, I suggest that a position assigned to moving objects by the visual system is determined by the success or failure of the animal that uses that information during action; if the behavior produced by the animal on the basis of position information is successful, contributing to the animal's survival, then the position is correct (and it is the same as that of the end-effector engaged in the production of action; see sect. R6.5). Otherwise, it is incorrect. McFarland claims that compensation may be unnecessary, and phase-alignment between different signals may be sufficient. Phase-alignment may solve one problem, but it raises another. Even if phase-alignment causes fusion of the to-be-aligned signals (say, vision and touch) the question still remains: How is the animal able to successfully interact with a visuo-tactile object, for example, an insect moving rapidly on the skin, given that both vision and touch give delayed position information?

Both McFarland and Cutting describe an experiment on audiovisual speech where the auditory and visual (lip movements) signals are asynchronous. The findings are that observers fail to detect moderately large asynchronies. Cutting suggests that the examples of large tolerances show that compensation is not necessary. I have an alternative interpretation of these interesting findings: The nervous system reduces discrepancies between two (or more) neural representations that belong to the same world event. This function of VP is outlined in the target article (see section 9.1. “Coordination of neural representations”). A previous study has shown that the brain can use visual distance to calibrate the simultaneity of auditory and visual signals of audio-visual objects (Sugita & Suzuki Reference Sugita and Suzuki2003). Thus, the asynchrony between auditory speech and its visual signal is undetected not because there is a wide range of tolerance, but because the brain actively coordinates the auditory and visual signals that likely belong to the same physical event.

There is a more fundamental aspect of this coordinating influence. There are many neurons in the CNS that receive convergent input from more than one modality, for example, audition and vision. Consider the consequences of unchecked lags between auditory and visual channels from a neurophysiological perspective. As vision is slower than audition, differential processing delays would mean that the input to bimodal neurons will be asynchronous, and for moving audiovisual objects, the two signals will be spatially offset in location maps. But, most importantly, an asynchrony would mean that the convergent inputs from moving objects would repeatedly fail to activate such bimodal neurons. This has implications not just for perception but also for the survival of such neurons. One possible solution could be a delay introduced in the faster modality; however, this would mean that early warning from an audiovisual object would cause the animal to respond with the latency of the slower modality. A much more adaptive solution would be for the nervous system to extrapolate the slower modality.

Cutting argues that compensation for delays is not necessary for fine-timed interceptions and claims that behaviors in sports are better understood in terms of skill acquisition over many years (combined with a genetic predisposition). There is no doubt that skill and genetics are important in everyday interactions and crucial in modern-day sports. However, the suggestion that skill can render compensation for sensory delays unnecessary is probably wrong. Highly skilled actions have features in common with ballistic actions (such as saccades); in both cases, the role of sensory data is reduced/altered. Hence, one may argue that compensation for sensory input delays during such open-loop actions becomes less relevant. However, in extreme cases of skilled performance, one may conjecture that no sensory input, let alone compensated sensory input, is required. Consider the performance of a knife-throwing artist. With enough skill, the artist can throw knives blindfolded. However, it is known from research on deafferented patients, for example, that there are limitations to open-loop actions; while some aspects of open-loop actions seem unaffected others aspects are compromised (Sarlegna et al. Reference Sarlegna, Gauthier, Bourdin, Vercher and Blouin2006). So sensory data are required even in highly skilled acts, and compensation for sensory delays is important.

Wagman, however, denies the very existence of neural delays. He challenges the view that perception and action are information-processing tasks, and so transmission of information between perception-to-action is not even possible. I am happy to treat perception-action as a hyphenated quantity, and I agree that the wide conceptual gap between perception and action that is sometimes assumed to exist is unjustified. Furthermore, I agree that the relational variables (e.g., in the expanding retinal image) are the most relevant in guiding actions. However, I do not see how these positions lead to the conclusion that delays in perception and action do not exist. In any event, the issue of delays discussed at length in the target article is not one of delays between perception and action but, rather, between a physical event and its perception, and a physical event and an action directed at it. So it is not clear whether commentator Wagman is addressing the main issues raised there.

R4.2. The faster the better, but not too fast please!

Lin suggests that the target article overestimates the visual processing delays. He further claims that the need for compensation depends on the assumption that perception occurs beyond V1. I disagree. The requirement for compensation is virtually independent of which segment of the vision pathway is considered necessary and sufficient for visual perception. This is so because it is well established that visual delays are nontrivial, even at the first processing level; photoreceptor hyperpolarization peak to a bright flash occurs with a latency of about 50 msec. Many actions in high-speed ballgames require a temporal window that is narrower than photoreceptor delays (Tresilian Reference Tresilian1993). Delays quickly become large as they accrue along the vision pathway. Hence, there is no question of compensation being unnecessary for delays before V1.

Lin's suggestion that neural delays are optimal – they are long enough to allow sufficient computation time, but not so large so as to make the responses too slow – is interesting. Similarly, Changizi points out that contrary to the assumption that faster neural processes would always confer an advantage to the animal, there may be a trade-off between computation speed and complexity. Changizi makes the point with an insightful analogy between nervous systems and computers. Computation speeds have increased by several orders of magnitude in the past two decades, and yet we still have to wait for software to load. Clearly, modern programs are doing a lot more. This behavior (rather than instantaneous loading of programs that do a lot less) is selected for in the computer world. In the real world, some behaviors can be produced with little computation – and relatively quickly – whereas others require elaborate computations. The knee-jerk reflex, for example, requires a single synapse and can be produced within 40 msec. However, such responses are stereotyped and have a limited (though important) function. The plasticity, flexibility, and complexity characteristic of many primate behaviors requires significant processing and elaborate networks. At the level of perception, the issue of compensation for delays may arise precisely because complex and time-consuming computations (e.g., integration of retinal image size and retinal disparity carried out by cortical cells in the computation of size constancy) are required for accurate localization of moving objects in three-dimensional space.

However, there are two things to note here. First, there is no doubt that increased speed of transmission of unmodified information over long distances, for example by peripheral sensory-motor nerves, can only be beneficial. Second, there are several other types of trade-off that are relevant. One is speed and size of the nervous system. Although certain invertebrate nervous systems are capable of producing fast reactions (e.g., the escape reaction in the squid), these reactions are possible only by the virtue of large-diameter nerve fibers. Myelination of neurons in vertebrates results in signal transmission that is more than an order of magnitude faster without the need to support much larger fibers (and brains). Therefore, although all this speed may not translate into faster behavior of vertebrates, generally speaking there is certainly some speeding up. Thus, it may be correct that software-loading times have not gone down linearly with the increase in computation speed but there is some general speeding up of computers.

Perhaps the most significant type of trade-off is revealed by comparison of processing speeds for predictable versus unpredictable events. In animal behavior, unpredictable events are of great importance. For example, in predator–prey interactions, each animal attempts to make gains over the opponent by producing sudden changes in trajectory. Such behaviors involving high accelerations are energetically expensive, so the fact that they are common in nature underscores their importance; these behaviors exist because the opponent's nervous system takes time to process them. It is likely that the upper limit of how much delay the animal's visual system will tolerate depends on how quickly the animal can respond to unpredictable visual events. A brain that is slower because of its fancier computations would take a long time in communicating an important, unpredictable, event (such as the predator adopting an attack posture). Thus, slowness due to fancier computations that may allow for extrapolation farther into the future, as suggested by Changizi, may indeed have no cost for highly predictable events, but the resulting slowness in the registration of unpredictable events (that are presumably processed by the same brain structures) will confer a huge disadvantage.

R6.1. Implementation of Visual Prediction

I agree with Trautteur, Datteri, & Santoro [Trautteur et al.] that the first goal of the target article is to make a convincing case that prediction is a widespread function of the nervous system. This goal would be impossible to achieve without inclusion of visual processes – hence my focus on VP. Indeed, my hope is to reduce the opposition to VP that has existed in raging controversies over the past decade or so. How VP is implemented is the next exciting question, and I am happy that the authors think that there are some useful rough sketches in the target article. Toward this goal, I would like to ask: Might there be as much variety in mechanisms as there are time scales for performing different actions, as suggested by Balkenius & Gärdenfors? Stepp & Turvey outline a novel approach in their commentary that may show well-known phenomenon of neural synchronization (Sillito et al. Reference Sillito, Jones, Gerstein and West1994; Neuenschwander et al. Reference Neuenschwander, Castelo-Branco, Baron and Singer2002) in the new light of visual predictive phenomena. I think there is likely to be a great variety in mechanisms for prediction, only a few of which we have any details on.

R6.2. Early versus late compensation: Are internal models indispensable?

Stepp & Turvey help to not only broaden the basis of anticipation (to use their term) but also bring clarity to the issues. First, I agree with these commentators (and with Benguigui et al.) that it is wrong to assume that internal models are the basis of all anticipation; it was not my intention to confer this exclusivity to anticipatory mechanisms. Indeed, there are anticipatory neural mechanisms that do not fit the description of internal models (Barlow Reference Barlow1953), and Stepp & Turvey mention an interesting example from physics, termed anticipating synchronization. Indeed, anticipatory phenomena may lie in realms beyond animal behavior, cellular networks, or single cells, and may be present in physical systems (e.g., in semiconductor lasers with optical feedback). This suggests an approach that may be termed anticipatory behavior without internal models, which is reminiscent of a previous approach to intelligent behavior without internal representations (Brooks Reference Brooks1991).

Stepp & Turvey further point out that anticipating synchronization shares features (e.g., delayed feedback) with the behavior of animals and nervous systems. In anticipatory synchronization there is a driver and a slave. The observation is that the slave can synchronize with the future state of the driver. Is such a system neurophysiologically plausible? The answer is clearly, yes. There are many examples of synchronization of neural responses in the retina, lateral geniculate nucleus (LGN) and cortex (Neuenschwander et al. Reference Neuenschwander, Castelo-Branco, Baron and Singer2002). Consider the LGN-V1 driver–slave pair, with LGN as the driver and V1 as the slave. In this system, V1 sends delayed feedback to LGN. It is not difficult to imagine V1 activity synchronizing with the future state of LGN activity (Sillito et al. Reference Sillito, Jones, Gerstein and West1994). It is noteworthy, however, that the basis of prediction may be triplets of neural locations (instead of pairs) that are capable of producing zero time-lag synchronization (Chawla et al. Reference Chawla, Friston and Lumer2001).

Stepp & Turvey (and van de Grind) have suggested that I have portrayed VP as an exclusively high-level model. I disagree. In the target article, I argue that prediction is ubiquitous in the CNS, with both early and late mechanisms devoted to it; findings such as the flash-lag effect in the flash-initiated condition (Khurana & Nijhawan Reference Khurana and Nijhawan1995) and the color decomposition effect (Nijhawan Reference Nijhawan1997) would be difficult to explain in terms of high-level mechanisms. Furthermore, there are neurophysiological considerations that support early mechanisms for prediction (see sect. R4.1). Nonetheless, anticipation can manifest itself at “high” levels where one cannot escape internal models (Erlhagen Reference Erlhagen2003; Snyder Reference Snyder1999), and synchronization and phase alignment become inappropriate metaphors. I would like to reiterate the proposal in the target article: For a function as fundamental as prediction, the nervous systems uses as many tricks as possible, and the phenomena revealing this function may show up in many guises.

R6.3. Time from adaptation

Every time we make a saccadic eye-movement there is a discrete, but predictable, change in visual input. Mayo & Sommer describe a process in which adaptation from one input to the next could be used for the timing of the sequence of discrete stimuli. A modified version of such a timing mechanism could play a role in compensation for neural delays. The mechanism is revealed when two sequential stimuli are presented to neurons in the superior colliculus and the frontal eye field. For inter-stimulus intervals of less than 400 msec, the neural response magnitude to the second stimulus varies precisely with the delay between the first and the second stimulus.

Mayo & Sommer have wondered how such a mechanism could be adapted to compensate for delays. I speculate on one possibility for estimating delays involving three visually responsive neural sites (A, B, and C). Sites A and B are hierarchically organized, with site B farther away from the retina than site A. Site C receives input from both A and B. If a single stimulus is presented, neurons in site A will respond before those in site B. Thus, a single world event will lead to two successive neural events that are temporally separated by the delay between site A and site B. The delay between the responses of site A and site B can be estimated from the adaptation level (reduced response) of neurons in site C to the input from site B (second input). This would be one way in which the visual system could monitor its own delays, and not just delays between two external events.

Sheth & Wu describe a neural activity profile in a topographical (cortical) map generated by the movement of an object. This profile includes the neural activity correlated with the object's distant past, immediate past, current, and immediate future positions; the locus of maximum cortical activity in the map represents the “current” location of the object. These commentators point out that the neural mechanisms leading to this profile are basic to all neurons. However, there is another basic neural mechanism that the commentators have missed, and that is spatio-temporal filtering, which has been invoked as one of the mechanisms leading to spatial extrapolation. One mechanism that Sheth & Wu do mention is neural adaptation. Mayo & Sommer make a clever proposal that adaptation could be used by neural networks in the timing of sequential stimuli, and perhaps in compensation of neural delays.

Consider again Sheth & Wu's statement that the maximum of neural activity in a topographic map represents the object's current location. There are multiple topographic maps in the brain, so the described profile of neural responses will be present in all of the maps stimulated by a moving object. However, as these maps are located at various distances from the receptor sheet (e.g., in the superior colliculus, cortex, etc.), the neural activity maxima representing the object's current position will correspond not to one but to many positions in space. Some of these maps (e.g., superior colliculus) serve mainly a motor function (e.g., eye-movements), but others might mainly serve perception. Input from two or more such maps to a target site that produces an average would be required to eliminate the multiplicity in perceived positions. With little modification, an extrapolation process could achieve the same goal.

R6.4. Modeling forward-shift effects

Berzhanskaya presents a detailed computational model that might produce not only the flash-lag effect, but also several other well-known forward-shift effects. One may, however, ask why these computational models behave the way they do. Is there any functional significance to this behavior of the models, assuming that natural visual systems behave like this? Towards this, then, one may wonder whether prediction versus emergent property of directional selective mechanisms, as the commentary title suggests, is the correct contrast to draw. Consider an analogy: human photoreceptors are most sensitive to light around the yellow-green part of the spectrum, despite the fact that blue photons are more energetic. One might explain this in terms of the photochemical interactions and the neural networks tuned to produce a maximum response to yellow-green light. This answer is correct and suffices for certain purposes. However, the fact becomes more meaningful when one looks deeper into the cause. The facts that maximum energy in the solar spectrum is in the yellow-green region, and that the absorption coefficient of water, which the human eye contains, is smallest precisely over the range for which human vision is most sensitive, give a better understanding of the peak sensitivity in the yellow-green part of the spectrum (Nunes Reference Nunes2003). Likewise, the psychophysical phenomena or networks that produce forward shifts would be understood better in the context of neural delays and prediction.

Berzhanskaya has focused on direction selectivity. It is worth mentioning that the flash-lag effect has been observed in retinal ganglion cells of the rabbit and salamander that are directionally insensitive. Another point to note is related to the commentators's explanation of the flash-lag effect in terms of shift in position of the flash due to trailing inhibition left behind by the moving object. This predicts that the flash should be mislocalized relative to stationary objects in the direction opposite to object motion. We have found that flash mislocalization in the direction opposite to motion may contribute, however, it is not sufficient to explain the full flash-lag effect (Shi & Nijhawan, under review).

R6.5. Modifying perceptions

Balkenius & Gärdenfors note that the task for predictive sensory-motor processes is more complex than is portrayed by the target article, and, in particular, that the target article underestimates the contribution of feedback from past interactions in creating a compensated percept of the moving object. I agree with the commentators. This is indeed an omission, which I will attempt to rectify here.

Balkenius & Gärdenfors propose that predictions concerning a moving object's visual position result in an anticipated now, but it is not known until later whether the prediction was correct or incorrect. In the target article, I say that: “during a catch, the visual system modifies the perceived position of the ball so that it matches the position of the moving hand.” I now elaborate on this with the example of interceptive action directed at a moving object. A successful catch implies that the animal's CNS was able to issue motor commands to send the hand to the correct location. Now the question is: To which physical position in space did the visual neural representation of the moving object correspond at the time of interaction? Feedback (vision, touch, and sound) from the results of the action helps in answering this question for the CNS. The CNS uses the motor-sensory representation (motor command and proprioception) of the hand as a reference to which the visual representation is compared. Successful interactions lead to integration of hand and visual positions, and the moving object is assigned the same visual position as the hand. Helmholtz first suggested that the visual position code (local sign) for an object could be trained by the motor position code when an end-effector, for example, the subject's moving hand, stimulates the subject's own retina. There is evidence that cells in the posterior parietal cortex respond to visual stimulation resulting from the movement of limbs during action (Mountcastle et al. Reference Mountcastle, Motter, Steinmetz, Duffy, Edelman, Gall and Cowan1984). Because the feedback interactions occur in a plastic neural environment, the error between spatial position of the visual object and the hand is removed over multiple successful interactions.

R6.6. Extrapolation in time or space?

Zago & Lacquaniti propose that extrapolation in the domain of time rather than space would be more useful for the visual system. They argue that time-extrapolation would be better adapted to handling moving objects in nature, as these objects rarely travel at constant velocities. However, a question immediately arises: Is a time-extrapolated input the same as a reduced latency one? (This of course raises the question of how this is different from the differential latency accounts by Whitney & Murakami [Reference Whitney and Murakami1998] and Purushothaman et al. [1998]). The main problem with time-extrapolation, as Zago & Lacquaniti recognize, is one of finding a mechanism. Again, if the suggestion is analogous to latency reduction, then this can be achieved by neural facilitation along the motion path (Grzywacz & Amthor Reference Grzywacz and Amthor1993). However, this can only be achieved on the basis of lateral interaction, which is again a spatial process. Mayo & Sommer discuss a purely temporal mechanism based on neural adaptation, which is reliable and unaffected by trial-to-trial variations in response delays. It remains to be seen whether such a mechanism would have the necessary speed to produce the interception of accelerating objects, as has been demonstrated by Lacquaniti and colleagues. A time-extrapolation could supplement the task of prediction, and work cooperatively with space-extrapolation to yield a more robust system than each type of extrapolation alone would.

Cunningham suggests that in addition to spatial compensation for delays, there should also be temporal compensation; and in addition to compensation for delays for continuous events, there should also be compensation for discrete events. There are perhaps two ways of describing temporal compensation. One is similar to whatz Zago & Lacquaniti might call time-extrapolation. This may be called absolute temporal compensation, as it affects delays between a physical event and its registration. The other is relative temporal compensation, where the nervous system actively coordinates sensory signals to compensate for temporal asynchronies between modalities (see commentary by Cutting).

Relative temporal compensation is what Cunningham and colleagues have shown in their interesting experiments. Let us consider absolute temporal compensation further. First, there is a form of absolute temporal compensation, as demonstrated by experiments on finger tapping to auditory tones. Within limits, human subjects can predict repetitive discrete tones and put their finger taps just in advance of the tones (Mates et al. Reference Mates, Muller, Radil and Poppel1994). This is a good example of where the sensorimotor system must have information about the actual time of an external event. But can absolute temporal compensation occur for perception (as opposed to behavior such as finger taps)? Can the visual system generate the percepts of repetitive flashes, for example, simultaneously with the actual flashes? The answer would appear to be no, and this is where spatial and temporal compensations differ. The claim of spatial compensation is that it can put the percept of the moving object closer to (or even ahead of) the actual position of the moving object. In contrast, temporal compensation cannot, it seems, put the perceptual event close to the time of the actual event.

I fully endorse the amendment suggested by Cunningham that: “The goal of visual prediction is to use priors acquired from both previous experience and the currently unfolding visual stimulus to create a perceived state of the world that matches, as far as possible, the actual state of the world.”

R6.7. Mental extrapolation and (not “or”) visual extrapolation

Kerzel & Müsseler suggest that sensorimotor prediction and mental extrapolation, as opposed to visual extrapolation, can overcome perceptual latencies. There is no doubt that sensorimotor prediction is an important, and highly flexible, contributor to successful behavioral acts. The role of mental extrapolation in the context of flash-lag effect is, however, not as clear as these commentators propose. Kerzel & Müsseler claim that the missing predictive-overshoot in the flash-terminated condition opposes the visual extrapolation account, and they invoke mental extrapolation in explaining the forward-shift of fading moving objects (Maus & Nijhawan Reference Maus and Nijhawan2006). However, it is not clear how mental extrapolation escapes this very criticism. Why does mental extrapolation not lead to an overshoot in the flash-terminated condition?

Mental extrapolation falls in the general category of phenomena such as mental imagery and mental rotation. Researchers have investigated the neural basis of mental imagery. One of the key findings is that mental imagery tasks engage the primary visual cortex (Kosslyn & Sussman Reference Kosslyn, Sussman and Gazzaniga1994). In addition, Kosslyn and colleagues have found a number of similarities between mental imagery and visual perception, such as the topographic nature of both representations (Kosslyn et al. Reference Kosslyn, Thompson, Kim and Alpert1995). Thus, the existence of mental extrapolation would predict the existence of visual extrapolation. Kerzel & Müsseler's proposal that mental extrapolation exists but visual extrapolation does not is unparsimonious.

I claim that the task of mental extrapolation is not to solve the problem of neural conduction delays, but rather, it is to determine when a moving object, occluded by another object, will reappear (Wexler & Klam Reference Wexler and Klam2001). In the case of continuous sensory input from a moving object, the task of mental extrapolation is to determine the object's future position. The task of visual extrapolation is to use sensory input to determine the object's current position (after compensating for visual conduction delays). In past studies, Kerzel and colleagues have used either probe stimuli presented after a retention interval, or pointing movements, and so in effect asked for the remembered final position of the moving target. In flash-lag experiments, or in the task used by Maus and Nijhawan (Reference Maus and Nijhawan2006), participants make an online perceptual judgment comparing the position of the moving target to a flash or to a static probe. Although obviously the observer's response is given after the visual offset, the judgment is based on simultaneously visible stimuli. It is likely that the two experimental methods differentially engage mental and perceptual extrapolation. In this context, it is interesting to note that the forward-shift effect of the fading moving object observed by Maus and Nijhawan (Reference Maus and Nijhawan2006) is 175 msec, which is a much larger shift than the typical flash-lag effect of 80 msec. It is possible that this is a cumulative effect of both visual and mental extrapolation.