The target article presents an interesting analysis of visual prediction, and we fully agree with Nijhawan that predictions need to be made at different levels of the sensory-motor processing. However, Nijhawan's article severely underestimates the complexity of anticipation in the sensory-motor system. In particular, it ignores the important question of how an anticipatory system can tune its predictions to internal and external delays as a result of experience. Consider the example of catching a ball. It is not sufficient that “the visual system modifies the perceived position of the ball so that it matches the position of the moving hand,” as Nijhawan suggests (sect. 10, para. 3). Instead, such a task involves at least the following components that can be divided into a visual pursuit and a catch component. Even to just visually focus on the ball, its trajectory needs to be anticipated (Balkenius & Johansson Reference Balkenius and Johansson2007). Since visual processing and the eye movements following it are not instantaneous, it is necessary to predict where the ball is right now to be able to fixate it while it is moving. We call this the “anticipated now” because any sensory code that is synchronous with an external state must be based on anticipation.

The predictions resulting in the anticipated now may or may not be correct, but there is no way for the organism to correct these predictions until at a later time, when the true sensory input becomes available. At this time, it is possible to adapt the earlier predictions to the actual sensory input, something that requires that the earlier anticipation, as well as the sensory information used for it, should still be available. This implies that at every moment, the sensory input is used both to anticipate the future and to adapt earlier predictions, but because of processing delays, it cannot be used to code for the current state of the external world. Similarly, eye movements cannot be based on the anticipated now, but must be controlled by the anticipated future. Looking at a moving object therefore requires that the organism simultaneously maintain sensory information at five different time-scales: the current sensory input, the anticipated now and future, and previous predictions of the now and the future. By combining information at the different time-scales in an appropriate way, it is possible both to change the currently anticipated now and to make future predictions more accurate. The most appropriate way to model the combination of time-scales is presumably to use feed-forward models from control theory (Grush Reference Grush2004; Kawato Reference Kawato1999). The predictions can be generated with different Kalman filters. The Kalman gains of the scales can be weighted in different ways, depending on the experience of the organism.

Assuming the gaze system is correctly tuned, via some feed-forward mechanism, the temporal unfolding of the ongoing interaction with the visual target contains the information needed to predict the location of the ball in the future; but the task for the hand is not to move to any arbitrary point along the predicted trajectory of the ball. Instead, the sensory-motor system must direct the hand to the location where the ball will be once the motor command to reach that location has been executed. This introduces an additional type of complexity, since the time in the future when the hand will catch the ball depends on properties of the arm and hand as well as on the ball. Although this is strictly also true for eye movements, the physical lag of the system becomes more critical for arm movements.

The properties of the flash-lag effect become perfectly sensible within this framework. Because unexpected events cannot become part of the anticipated now until after a processing delay, they will be perceived as lagging any predictable event. Moreover, since point events are perfect predictors of themselves, an adaptive system will learn to let them replace any prediction based on prior information.

In the flash-terminated condition, the flash becomes part of the anticipated now at the same time as the detection of the disappearance of the target. The flash and the disappearance of the target should therefore be perceived as simultaneous and occurring at the physical location where these two events actually take place. Once this information is received, the best prediction is that the moving object disappeared where it was at the time of the flash. This does not mean that the movement of the object is not extrapolated; it only suggests that that extrapolation is replaced by better information when it is available. Although it is possible that biased competition plays a role in this process, it may be sufficient to assume that a system that adapts its predictions to actual information will learn to behave in this way.

It is clear that the number of different time-scales that are necessary in the brain is much larger than the few described here. Given that substantial delays influence all processing in the brain, it appears necessary to compensate for these by predictive mechanisms at all levels. This suggests that predictive abilities should be necessary at the level of individual neurons – or at least local circuits. The ongoing dynamical interaction among different parts of the brain is not too different from the brain's interaction with the external world.