Present cognitive science is characterized by a dichotomy separating sensory and motor domains. This results in a perceived gap between perception and action and is mirrored in leading theories of cognition. For illustration we consider the visual neurosciences, a paradigmatic field for the investigation of sensory processes. A discourse given by standard textbooks depicts a world external to the agent, with a set of pre-established attributes and objects. Sensory processing starts with transmitting these attributes by low-level neurons to subsequent stages. There, more elaborate computations extract patterns of stimulus features and objects. Up to this point, processing focuses on a veridical representation of the external world, serving for later decisions and actions. We argue in favor of a radical change of this view, assigning a central role to predictions of sensory consequences of one's own actions and thereby eliminating the strict separation of sensory and motor processing.

In the target article, Andy Clark beautifully describes the central role of predictions in sensory processing. We endorse this view – yet two complementary aspects are needed. First, predictability of sensory signals serves as a normative principle guiding sensory processing and as a boundary constraint in the selection of information to process. Second, predictions are performed only in the context of the agents' action repertoire (König & Krüger Reference König and Krüger2006). These two specifications have crucial implications.

The information content of the primary sensory signal is enormous, and extraction of information without further constraint is an ill-posed problem. However, it is not the task of the sensory systems to process all possible details, and a reduction of information is paramount. Even in simple model systems, taking into account a limited behavioral repertoire converts demanding sensory processing into a tractable problem (Wyss et al. Reference Wyss, König and Verschure2004). Applying the normative principle of predictability generalizes this idea and serves as a selection criterion for features to process and variability to ignore. Indeed, within the hierarchy of the visual system, neuronal response properties are invariant to more and more parametric changes of the sensory input (Tanaka Reference Tanaka1996). Even category learning at higher levels of the visual system can be interpreted within this framework. The commonalities between different instances of the same category relate to similar sensorimotor patterns generated by the interaction with these “objects.” Finally, actions are directly related to the agent's survival and thereby processing features that change predictably, given chosen actions, are more relevant than those that do not. Hence, processing of sensory signals is guided by the relevance for behavior, and relevance is expressed by the ability to predict sensory changes contingent on the own action repertoire.

A paradigm is based on the active interpretation of incoming sensory information such that it makes sense for the agent. Hence, it is intended to replace a passive representationalist view. In such a paradigm, the predicted future state of the world is important insofar as it interacts with own actions and variables of importance are co-determined by the action repertoire. A demonstration of the integration of new sensory information (magnetic north) that is co-determined by own movements (yaw-turns) is given by the feelSpace project (Kärcher et al. Reference Kärcher, Fenzlaff, Hartmann, Nagel and König2012; Nagel et al. Reference Nagel, Carl, Kringe, Märtin and König2005). Comparing different species, for example, cat and human, with similar visual input (Betsch et al. Reference Betsch, Einhäuser, Körding and König2004; Einhäuser et al. Reference Einhäuser, Moeller, Schumann, Conradt, Vockeroth, Bartl, Schneider and König2009), the remarkable differences in the sensory hierarchy appear to be at odds with a passive representationalist view and await an explanation. Here, differences in behavioral repertoire offer themselves. Pointedly, we speculate that the huge action repertoire of humans, due to, for example, opposable thumbs, might foster the illusion of a veridical perception of the world. It has been emphasized early on that cognitive and motor capabilities develop in parallel and mutual dependence (Piaget Reference Piaget1952). To grow up means to harden specific action routines, on the one hand, but to lose the bulk of alternative action capabilities and cognitive flexibility, on the other hand. Furthermore, a large variability of perceptual interpretation of identical physical stimuli is found between humans of the same culture area as well as between different cultures (Segall et al. Reference Segall, Campbell and Herskovits1963). A critical view of our own culture reveals many aspects that serve to increase the reliability of predictions. In summary, agents with identical sensory organs but different action repertoires might have very different views of the world.

Is the concept of normative principles plausible in view of our knowledge of cortical networks? Neuronal computations are constrained by properties of the brain in the form of number of neurons and synapses, and space and energy consumption. The latter has served as an argument for sparse coding – that is, low mean activity at constant variance of activity (Barlow Reference Barlow and Rosenblith1961). The insight that receptive fields of simple cells in primary visual cortex form such an optimally sparse representation of natural images drastically increased interest in normative models (Olshausen & Field Reference Olshausen and Field1996; Simoncelli & Olshausen Reference Simoncelli and Olshausen2001). Properties of the second major neuron type in primary visual cortex, complex cells, can be understood along similar lines as optimizing stable representations (Berkes & Wiskott Reference Berkes and Wiskott2005; Körding et al. Reference Körding, Kayser, Einhäuser and König2004). Importantly, both optimization principles can be easily implemented by recurrent connectivity within a cortical area (Einhäuser et al. Reference Einhäuser, Kayser, König and Körding2002). Hence, existing normative models of the early visual system are plausible in view of anatomical and physiological data.

A critical test of the concept will be the application well beyond processing in the primary visual cortex. The step from sparseness and stability to predictability as an optimization principle requires critical extensions. Phillips et al. (Reference Phillips, Kay and Smyth1995) put forward a very promising proposal: Coherent infomax selects and coordinates activities as a function of their predictive relationships and current relevance. The relation of this approach (see Phillips' commentary in this issue) to the free energy principle (Friston Reference Friston2010) and optimal predictability (König & Krüger Reference König and Krüger2006) has to be investigated. These developments hold the promise to apply to “higher” cognitive functions as well as giving rise to a true theory of cognitive science.