Scientific progress depends on our ability to formulate clear hypotheses that can be experimentally tested (i.e., models). The goal is to be able to explain why the data are what they are, which goes beyond a mere description of the data. Optimal inference (Helmholtz Reference Helmholtz1867) as a general hypothesis of perception has served the community extremely well in that regard, in particular in combination with the Bayesian framework (Knill & Richards Reference Knill and Richards1996). It provided quantitative but nonetheless intuitive explanations for many fundamental characteristics of perception, such as how sensory information from different sources is combined (Ernst & Banks Reference Ernst and Banks2002), how prior information affects the percept (Körding & Wolpert Reference Körding and Wolpert2004), and how stochastic choice behavior naturally emerges from a process of probabilistic inference (Stocker & Simoncelli Reference Stocker and Simoncelli2006a). That some of these specific models are not universally valid (the listed examples of “suboptimalities” in Rahnev & Denison's [R&D] article) lies in the iterative nature of scientific progress: Parsimonious models must be successively refined as new data demand modifications. It seems rather foolish to question a very successful general hypothesis because specific assumptions of a particular model implementation turn out to be simplistic. We are far from having exhaustively explored the optimality hypothesis and therefore should not abandon it lightly; I elaborate on this in the following:
The discussion is premature. “Optimal” per definition refers to best possible with respect to some given limitations/constraints. What separates an “optimal observer” from an “ideal observer” is that the latter only considers limitations in terms of information provided to the observer and thus is well defined in an experimental setting, whereas the former also includes constraints that are internal to the observer. Because we just started to explore these constraints (such as limited representational resources), we are not yet in a position to make with any confidence a general assessment of whether perception is optimal or not. Hence, we should simply abstain from drawing any premature conclusions.
Optimality is a very valuable, normative hypothesis. The general hypothesis has proved extremely valuable in deriving models of perceptual decision making, in particular (but not only) in combination with the Bayesian formalism. Its normative nature allows us to formulate with relative ease an observer model for any specific task and thus to possess a quantitative model hypothesis before actually running the experiment and knowing the data. This is a substantial advantage as it empowers us to not only design experiments that are most efficient in validating the model, but also to cross-validate the model by making specific predictions for a new task based on the model parameters determined from data of a previous task. It is also important to realize that without optimality assumption, the Bayesian formalism would have been unlikely adopted by the community over the last 20 to 30 years, a formalism that undeniably was very successful by any rational metric. Bayesian decision theory and the optimality assumption are in many ways synonymous; without the latter, the former is not meaningful. Finally, optimality is by no means an arbitrary hypothesis but ultimately directly follows from the theory of evolution: A system's actions and behaviors are aimed to perform in the best way possible in order to optimize the chances of survival and reproduction in a competitive environment. Briefly, the optimality assumption is a well-supported, very useful assumption.
Is there an alternative? “To deny that we reason in a Bayesian way is to assert that we reason in a deliberately inconsistent [i.e., random] way” (Jaynes Reference Jaynes1957/2003, p. 133). Clearly, Jaynes did not think that there was an alternative. But even with a more measured view, it is difficult to conceive of such a possibility. As R&D rightly state, “suboptimal” is not an alternative: Simply rejecting the null hypothesis is not a hypothesis, a common fallacy far too often encountered in the psychological sciences. Any alternative hypothesis of equal value must have a normative character (i.e., it must allow us to formulate quantitative models for specific perceptual decision-making tasks). Not only that, but it also must explain why under some conditions perceptual behavior is seemingly optimal and under others it is not. R&D's proposed “standard observer model” is as fuzzy as they described it (could be a “bag of tricks” or a “neural network”; sect. 5.2, para. 1) because they have no idea what alternative hypothesis it should represent – there simply is no equivalent alternative to the optimality assumption at the moment.
So, how do we best move forward? Let us simply follow proper scientific procedure: The current evidence in favor of optimality far outweighs the evidence in favor of any other potential hypothesis (whatever that might be). This does not mean that the experimental evidence suggesting “suboptimal” behavior should be ignored, on the contrary. But we should not prematurely abandon the optimality assumption either. Rather we should continue probing the general normative hypothesis that has been so good to us and try to refine and extend it to make it fit with new experimental data that require so. Recent work from my laboratory may serve as an example for this approach: We have noticed that perceptual biases (e.g., in perceived visual orientation) are frequently away from the peak densities of the expected prior distribution, which contradicts the predictions of a traditional optimal Bayesian observer (de Gardelle et al. Reference de Gardelle, Kouider and Sackur2010; Tomassini et al. Reference Tomassini, Morgan and Solomon2010). But rather than claiming suboptimal behavior and calling it a day, we realized that the traditional Bayesian observer model relies on implicit assumptions that actually may be incorrect (in this case the noise distributions). Indeed, we showed that if we add an additional constraint to the observer model – namely, that the observer's representational resources are limited and must be used efficiently (the efficient coding hypothesis [Barlow Reference Barlow and Rosenblith1961], another optimality assumption) – then such “suboptimal” behavior indeed turns out to be perfectly optimal (Wei & Stocker Reference Wei and Stocker2015). Not only that, but the additional constraint also allowed us to discover a new perceptual law, describing a functional relationship between perceptual bias and discrimination threshold (Wei & Stocker Reference Wei and Stocker2017). Many of the “suboptimalities” that R&D list (in their Table 1) can be thought of as observer-related constraints and limitations, and it seems more likely than not that eventually, they can all be described within an optimal model. There is much potential in using the optimal (Bayesian) observer model as our well-defined standard model and improving and extending it with additional constraints that we discover based on computational, psychophysical, and physiological considerations.
Eventually, however, I absolutely agree with R&D that we should not waste our energy with dogmatic battles (in particular because the topic of these battles seems not well defined) but rather focus on “building and testing detailed observer models that explain behavior across a wide range of tasks” (abstract) and “capture all the systematic weirdness of human behavior rather than preserve an aesthetic ideal” (sect. 6, para. 1). We are scientists, and as such, we should ideally value and judge different models solely based on their ability to account for and rightly predict the full richness of the data. However, in putting us in a position to do this, the value of the optimality hypothesis is currently unrivaled.
Scientific progress depends on our ability to formulate clear hypotheses that can be experimentally tested (i.e., models). The goal is to be able to explain why the data are what they are, which goes beyond a mere description of the data. Optimal inference (Helmholtz Reference Helmholtz1867) as a general hypothesis of perception has served the community extremely well in that regard, in particular in combination with the Bayesian framework (Knill & Richards Reference Knill and Richards1996). It provided quantitative but nonetheless intuitive explanations for many fundamental characteristics of perception, such as how sensory information from different sources is combined (Ernst & Banks Reference Ernst and Banks2002), how prior information affects the percept (Körding & Wolpert Reference Körding and Wolpert2004), and how stochastic choice behavior naturally emerges from a process of probabilistic inference (Stocker & Simoncelli Reference Stocker and Simoncelli2006a). That some of these specific models are not universally valid (the listed examples of “suboptimalities” in Rahnev & Denison's [R&D] article) lies in the iterative nature of scientific progress: Parsimonious models must be successively refined as new data demand modifications. It seems rather foolish to question a very successful general hypothesis because specific assumptions of a particular model implementation turn out to be simplistic. We are far from having exhaustively explored the optimality hypothesis and therefore should not abandon it lightly; I elaborate on this in the following:
The discussion is premature. “Optimal” per definition refers to best possible with respect to some given limitations/constraints. What separates an “optimal observer” from an “ideal observer” is that the latter only considers limitations in terms of information provided to the observer and thus is well defined in an experimental setting, whereas the former also includes constraints that are internal to the observer. Because we just started to explore these constraints (such as limited representational resources), we are not yet in a position to make with any confidence a general assessment of whether perception is optimal or not. Hence, we should simply abstain from drawing any premature conclusions.
Optimality is a very valuable, normative hypothesis. The general hypothesis has proved extremely valuable in deriving models of perceptual decision making, in particular (but not only) in combination with the Bayesian formalism. Its normative nature allows us to formulate with relative ease an observer model for any specific task and thus to possess a quantitative model hypothesis before actually running the experiment and knowing the data. This is a substantial advantage as it empowers us to not only design experiments that are most efficient in validating the model, but also to cross-validate the model by making specific predictions for a new task based on the model parameters determined from data of a previous task. It is also important to realize that without optimality assumption, the Bayesian formalism would have been unlikely adopted by the community over the last 20 to 30 years, a formalism that undeniably was very successful by any rational metric. Bayesian decision theory and the optimality assumption are in many ways synonymous; without the latter, the former is not meaningful. Finally, optimality is by no means an arbitrary hypothesis but ultimately directly follows from the theory of evolution: A system's actions and behaviors are aimed to perform in the best way possible in order to optimize the chances of survival and reproduction in a competitive environment. Briefly, the optimality assumption is a well-supported, very useful assumption.
Is there an alternative? “To deny that we reason in a Bayesian way is to assert that we reason in a deliberately inconsistent [i.e., random] way” (Jaynes Reference Jaynes1957/2003, p. 133). Clearly, Jaynes did not think that there was an alternative. But even with a more measured view, it is difficult to conceive of such a possibility. As R&D rightly state, “suboptimal” is not an alternative: Simply rejecting the null hypothesis is not a hypothesis, a common fallacy far too often encountered in the psychological sciences. Any alternative hypothesis of equal value must have a normative character (i.e., it must allow us to formulate quantitative models for specific perceptual decision-making tasks). Not only that, but it also must explain why under some conditions perceptual behavior is seemingly optimal and under others it is not. R&D's proposed “standard observer model” is as fuzzy as they described it (could be a “bag of tricks” or a “neural network”; sect. 5.2, para. 1) because they have no idea what alternative hypothesis it should represent – there simply is no equivalent alternative to the optimality assumption at the moment.
So, how do we best move forward? Let us simply follow proper scientific procedure: The current evidence in favor of optimality far outweighs the evidence in favor of any other potential hypothesis (whatever that might be). This does not mean that the experimental evidence suggesting “suboptimal” behavior should be ignored, on the contrary. But we should not prematurely abandon the optimality assumption either. Rather we should continue probing the general normative hypothesis that has been so good to us and try to refine and extend it to make it fit with new experimental data that require so. Recent work from my laboratory may serve as an example for this approach: We have noticed that perceptual biases (e.g., in perceived visual orientation) are frequently away from the peak densities of the expected prior distribution, which contradicts the predictions of a traditional optimal Bayesian observer (de Gardelle et al. Reference de Gardelle, Kouider and Sackur2010; Tomassini et al. Reference Tomassini, Morgan and Solomon2010). But rather than claiming suboptimal behavior and calling it a day, we realized that the traditional Bayesian observer model relies on implicit assumptions that actually may be incorrect (in this case the noise distributions). Indeed, we showed that if we add an additional constraint to the observer model – namely, that the observer's representational resources are limited and must be used efficiently (the efficient coding hypothesis [Barlow Reference Barlow and Rosenblith1961], another optimality assumption) – then such “suboptimal” behavior indeed turns out to be perfectly optimal (Wei & Stocker Reference Wei and Stocker2015). Not only that, but the additional constraint also allowed us to discover a new perceptual law, describing a functional relationship between perceptual bias and discrimination threshold (Wei & Stocker Reference Wei and Stocker2017). Many of the “suboptimalities” that R&D list (in their Table 1) can be thought of as observer-related constraints and limitations, and it seems more likely than not that eventually, they can all be described within an optimal model. There is much potential in using the optimal (Bayesian) observer model as our well-defined standard model and improving and extending it with additional constraints that we discover based on computational, psychophysical, and physiological considerations.
Eventually, however, I absolutely agree with R&D that we should not waste our energy with dogmatic battles (in particular because the topic of these battles seems not well defined) but rather focus on “building and testing detailed observer models that explain behavior across a wide range of tasks” (abstract) and “capture all the systematic weirdness of human behavior rather than preserve an aesthetic ideal” (sect. 6, para. 1). We are scientists, and as such, we should ideally value and judge different models solely based on their ability to account for and rightly predict the full richness of the data. However, in putting us in a position to do this, the value of the optimality hypothesis is currently unrivaled.