The environments in which an organism lives and the tasks it performs to survive and reproduce shape the design of its perceptual, cognitive and motor systems through evolution and experience. This is an obvious statement, but it implies four fundamental components of research we must have if we are going to gain a deep understanding of these biological systems, and of how to design similar artificial systems: (i) methods for identifying and characterizing natural tasks, (ii) a science devoted to measuring and characterizing natural environments and stimuli (iii) mathematical and computational methods for understanding how intelligent systems could use knowledge of environmental and motor constraints to perform natural tasks, and (iv) experimental methods and technology that allow rigorous measurement of behavioral and neural responses, either in natural tasks or in artificial tasks that capture the essence of the natural tasks. These components are at the core of natural systems analysis, a scientific approach that has been gaining momentum in recent years and is revolutionizing research in behavioral science, neuroscience, and computational science. The articles in this special issue exemplify this emerging scientific approach.
What are natural environments and tasks? Natural environments generally refer to the outdoor environments (forests, grasslands, mountains) in which our species and other species evolved, but they also include human-made indoor or outdoor environments. In humans, natural tasks refer to everyday tasks that we perform (or our ancestors performed) to survive, such as navigating through an environment, searching a crowd for a face or a voice that we know, picking fruits or vegetables from a plant, and making tools. These seem like simple tasks but they are extraordinarily complex and difficult, especially in natural environments, and thus behavioral scientists and systems neuroscientists have traditionally focused on simplified tasks carried out in laboratory environments with simple stimuli.
Although there is no substitute for carefully-controlled laboratory experiments, scientific progress can be held back by not having a detailed understanding of natural environments and tasks. As just one example, consider the task of finding a familiar face in a crowd. To perform this task the brain must find faces using peripheral vision, direct the eyes towards each of them, and then compare the features of that face with stored representations of the faces it has seen before. To do these things well requires detailed knowledge of the relationship between the images that are formed in the eye and the actual three dimensional faces that occur in the environment. This relationship is extremely complex because of the enormous variation in the images produced by exactly the same face from one occasion to the next. The sources of variation include the distance and orientation of the head, the lighting that forms shadows and shading on the face, the presence of other objects that may partially block the face, and so on. The brain has learned about all these sources of variation through evolution and experience and exploits this knowledge to obtain remarkably accurate perception, recognition and control of eye movements.
Unfortunately, we are still a long way from understanding how the brain accomplishes such tasks. A major reason for the slow progress may be the traditional scientific approach, which focuses on neural mechanisms and computational algorithms without first obtaining a detailed understanding of the relevant properties of the environment and the requirements of the task. In the traditional approach, the scientist thinks about the natural environment and task, generates hypotheses about neural mechanisms or useful algorithms, and then conducts controlled behavioral, neural or computational experiments to test those hypotheses. The weakness of this approach is that the hypotheses are generated from “thinking” about the environment and tasks rather than from principled measurements. To generate plausible hypotheses about how the brain finds familiar faces in a crowd we need to know the statistical variations in the three-dimensional shapes of faces, the statistical variations in natural lighting, the useful features for detecting faces in the periphery and for distinguishing one face from another, and so on, because it is precisely this kind of knowledge that the brain exploits to perform the task. It is difficult, maybe impossible, to generate truly plausible hypotheses for neural mechanisms without characterizing the relevant constraints and sources of variation that exist in the natural environment. The instrumentation and computational tools to rigorously characterize the properties of natural environments are now becoming available and are beginning to be usefully exploited, as demonstrated in this special issue and in other recent studies.
Analysis of task requirements and measurements of environmental properties are essential, but are only two components of natural systems analysis. Another crucial component is mathematical and computational analysis of how to exploit the measured environmental properties (e.g., scene statistics) to perform natural tasks. There are various approaches, but the most promising are Bayesian ideal observer theory, information theory, and optimal learning theory in computational neural networks. These mathematical tools make it possible to determine what the brain should compute to perform a task optimally, given particular environmental properties and biological constraints. This is crucial for understanding what environmental knowledge is important for performing a task and for specifying the computations a rational neural or artificial system should perform. Such mathematical analyses, some of which are described in this special issue, provide the proper starting point for generating principled hypotheses and are playing an increasingly important role in guiding behavioral and systems neuroscience. It is, of course, important to remember that the solution implemented by an organism for a given natural task also depends strongly on the organism's evolutionary history. Much of this history is represented implicitly by the biological constraints included when determining optimal performance. However, additional insight into perceptual and motor systems might be obtained by directly characterizing the evolutionary (and co-evolutionary) history of the organism.
The final crucial component of natural systems analysis is to develop and exploit technology that allows rigorous testing of hypotheses about neural mechanisms. The traditional scientific approach is to use simplified tasks (or in the case of neuroscience sometimes no task at all) and simple stimuli. This approach has been driven by the fact that it is impossible to test hypotheses rigorously without a high degree of experimental control. Nonetheless, to fully test hypotheses about how the brain performs natural tasks, it is necessary to conduct controlled experiments in which the organism is performing a natural task or at least performing a task that captures the essence of the natural task. The technology for doing this is developing rapidly. It is now possible to precisely measure the positions of the eyes, limbs and body in natural tasks; to measure the activity of single neurons or of whole populations of neurons in the brain while an animal is performing natural tasks; to create virtual environments containing visual, tactile, and auditory objects that can be precisely controlled and manipulated during the performance of natural tasks; and to create artificial agents that can test algorithms for performing natural tasks. As demonstrated in this special issue, this rapidly developing technology is making it possible to bring the rigor of precise experimental control to the domain of natural tasks.
The articles in this special issue are divided into groups based on the components of natural systems analysis they emphasize. Manning and Brainard, Wiltshut and Hamker, and Hansen and Gegenfurtner focus on characterizing natural scene statistics relevant for efficient encoding of sensory information in the early visual system. Efficient sensory coding can support a wide range of natural tasks. Manning and Brainard investigate the design of photoreceptor mosaics within a general Bayesian framework of image reconstruction. They show that there is a trade-off between monochromatic and dichromatic designs as a function of overall light intensity. The analysis offers a principled explanation for why we see color during the day but not at night. A number of different definitions of efficient coding have been proposed as principles that may have shaped the design of the early visual system. In a novel analysis, Wiltshut and Hamker use a Hebbian/anti-Hebbian neural network model to train local receptive fields given different definitions of efficient coding. They find that most existing definitions, when added to the model, increase the correlation of the trained spatial receptive fields with those that have been measured in monkey visual cortex. Hansen and Gegenfurtner discuss a new analysis of calibrated natural images and show, contrary to prior claims, that chromatic and luminance signals provide independent sources of information that can be combined to detect boundaries in scenes. These findings provide a new framework for the analysis of chromatic receptive fields in early stages of visual processing.
Land, Greenwald and Knill, and Hayhoe, Gillam, Chajka and Vecellio focus on measuring performance of humans in natural tasks and on characterizing the general kinds of stimulus information they use to perform those tasks. Land provides a review of data in support for the notion that tasks can have tremendous influence on the organization of complex action sequences, including the movement of the eyes. He brings up examples such as making a cup of tea, preparing sandwiches, music reading and portrait sketching. He emphasizes the observation that task-irrelevant objects are almost never fixated, which suggests that the control of fixation sequences cannot be understood from simple bottom-up salience of features. Greenwald and Knill studied how the use of monocular/binocular information of three-dimensional orientation is used in different visuomotor tasks: prehension and object placement. Their analyses reveal that subjects rely upon different sources of visual information depending on the task at hand. These findings emphasize the necessity of future studies of ‘natural tasks’ and how they influence early visual processing. Navigation is a basic task for all moving animals. Hayhoe and colleagues analyze the role of binocular vision in a navigation task where subjects have to walk over and around obstacles. They report that binocular vision, normally believed to assist in the fine manipulation of objects, has a strong effect on the efficiency of walking across obstacles.
Rothkopf and Ballard, Zhang, Abbey and Eckstein, Geisler and Perry, and Torralba focus on characterizing natural scene statistics relevant for specific natural tasks. Rothkopf and Ballard measured image statistics at the point of gaze while humans were navigating through a virtual environment. They find that feature properties at the point of gaze vary systematically depending on whether subjects are navigating toward or around an object. The results suggest that the relevant image statistics for natural tasks can change substantially with subtle changes in task goals. Zhang et al. derive the optimal linear receptive fields for detecting localized targets in natural images using a genetic learning algorithm. They find that these optimal linear receptive fields have an inhibitory surround that is similar to those of behavioral receptive fields estimated from saccadic search tasks in white noise. Their results offer a new hypothesis for the center-surround organization of receptive fields in the visual pathway. Geisler and Perry describe a new analysis of the statistics of pair-wise contour elements in natural scenes and derive an ideal Bayesian observer for a contour occlusion task. They find that human observers parallel ideal performance uniformly across the four-dimensional space defining the stimulus domain. The findings demonstrate a tight relationship between statistical properties of sensory signals and the mechanisms that process them. Torralba asks the question of how many pixels in a natural scene image are required to identify the general category of scene and the dominant objects within that scene. Remarkably, he finds that humans can accomplish this task with images that are only 32 x 32 pixels. The results support previous findings that very low resolution features are sufficient to identify the category of natural scene and that the context provided by the scene category is sufficient to identify otherwise unidentifiable objects within the scene.
Juricevic and Webster, and Maloney and Mamassian focus on the role and nature of plasticity in natural tasks. Juricevic and Webster use a computational analysis of calibrated natural images to explore the expected consequences on color appearance of long-term adaptation to different natural environments (e.g., arid vs. lush environments). Although appropriate data for detailed testing of the predictions is not yet available, their computational results imply that color vision should be strongly dependent on the specific natural environment that a given human inhabits. There are a number of perceptual and motor tasks where humans parallel the performance predicted by ideal Bayesian decision theory; however, it is unknown whether humans accomplish this with simple heuristics or by actually implementing the machinery of Bayesian inference. To address this fundamental question, Maloney and Mamassian propose novel experimental tests based on measuring the transfer of learning across related perceptual-motor tasks. The key insight of this approach is that if humans actually implement the machinery of Bayesian inference then they should be able to rapidly transfer their learning in one natural task to related tasks.
In conclusion, there has been a growing realization among researchers that perceptual and motor systems must reflect the natural tasks the organism performs, the properties of natural stimuli that allow performance of those tasks, certain relevant biological constraints, and the evolutionary history of the organism. Thus, the four components of natural systems analysis described earlier (and exemplified by the articles in this special issue) are certain to play an increasingly important role in behavioral science, systems neuroscience and computational science.
