Bayesian Rationality (Oaksford & Chater Reference Oaksford and Chater2007) nicely synthesizes the growing body of research. This approach offers promise to give principled explanation not only of human higher cognitive processes, but also much of perception and animal cognition. Is cognition best described by optimal models, or by adaptive but suboptimal heuristic strategies? Oaksford & Chater (O&C) explain that heuristic strategies approximate human syllogistic reasoning better than optimal Bayesian models. This commentary illustrates, in the context of information search, that it is frequently possible to be mistaken about what model is best motivated; and that naïve heuristic strategies can perform better than “optimal” models!

Consider the task of deciding which of two medical tests to order, assuming cost constraints only allow one test, to best diagnose a patient's disease. We assume here that the patient either has Disease 1 or Disease 2, with equal (50 %) prior probability, and that Test 1 and Test 2 are each either positive or negative. How should a diagnostician decide which test to order?

A great deal of cognitive psychological and statistical thinking (since I. J. Good's [Reference Good1950; Reference Good1975] work) claims that the optimal strategy is to conduct the test with highest expected Bayesian diagnosticity (expected weight of evidence) or expected log diagnosticity. [T1=p denotes that Test 1 is positive, T1=n that Test 1 is negative, D=d1 that the patient has disease 1, etc.] As shown in Figure 1, the expected Bayesian diagnosticity for Test 1 (i.e., its expected utility, as measured with Bayesian diagnosticity) is:

Figure 1.

and, as shown in Figure 2, its expected log diagnosticity is:

Figure 2.

However, many subjects follow the feature difference strategy (Nelson Reference Nelson2005; Skov & Sherman Reference Skov and Sherman1986; Slowiaczek et al. Reference Slowiaczek, Klayman, Sherman and Skov1992). This strategy involves calculating the absolute difference in feature likelihoods for each test; for example, as shown in Figure 3:

Figure 3.

and ordering the test with the highest fDiff.

Thus, Skov and Sherman and Slowiaczek et al. concluded that many subjects use a suboptimal heuristic strategy that is highly correlated with the optimal strategy. Remarkably, however, both the claims (1) that Bayesian diagnosticity (and/or log diagnosticity) are theoretically optimal, and (2) that the feature difference strategy only imperfectly approximates optimal behavior, are in disrepute.

Both expected Bayesian diagnosticity and expected log diagnosticity are poorly behaved as optimal models. To illustrate, suppose that Test 1 were positive in 99% of people with Disease 1, and in 100% of the people with Disease 2. Suppose further that Test 2 were positive in 1% of people with Disease 1, and 99% of people with Disease 2. Test 1 leads, on average, to 50.5% probability of identifying the correct disease; Test 2 leads, on average, to 99% probability of correctly identifying the true disease. Clearly, Test 2 would be more helpful than Test 1 to differentiate between the diseases. Yet diagnosticity and log diagnosticity maintain that Test 1 is infinitely more useful than Test 2! Both diagnosticity measures hold that any test that offers greater-than-zero probability of obtaining 100% certainty of the true disease is infinitely useful. This bizarre claim is not a desirable property of an “optimal” model. (In Nelson [2005; 2008] I discuss these and other theoretical flaws with the diagnosticity measures, and how redefining a single point cannot fix them.)

Better-motivated theoretical models of the value of information, such as information gain-KL distance (Lindley Reference Lindley1956; Oaksford & Chater Reference Oaksford and Chater1994), probability gain (error reduction; cf. Baron's 1981 talk at the Psychonomic Society Meeting, as cited in Baron Reference Baron1985), and impact (Klayman & Ha Reference Klayman and Ha1987, pp. 219–20; Nelson Reference Nelson, Oaksford and Chater2008; Nickerson Reference Nickerson1996; Wells & Lindsay Reference Wells and Lindsay1980) behave reasonably in this medical diagnosis scenario, and do not suffer from the diagnosticity measures' aforementioned theoretical flaws.

Does the feature difference strategy also approximate these better-motivated theoretical models? In fact, it exactly corresponds to impact! The highest fDiff feature also has the highest impact, irrespective of the prior probabilities of the diseases and the specific feature probabilities (Nelson Reference Nelson2005, footnote 2).

Closer analysis of the supposedly optimal theoretical models used by some experimenters, and the supposedly suboptimal heuristics used by some subjects, showed that the subjects' heuristic strategy corresponds to a normative model (impact) that is theoretically superior to the normative model that the experimenters had in mind! Put in the context of Marr's (Reference Marr1982) levels of analysis, consideration of subjects' behavior at the algorithmic level can inform thinking about the kinds of computational-level models (normative theories) that are most appropriate (also see Chater et al. Reference Chater, Oaksford, Nakisa and Redington2003; Cohen Reference Cohen1981).

Do all subjects use the feature difference strategy? No. As O&C discuss, the means with which information is presented is important. Different people use a variety of strategies, especially when environmental probabilities are presented in the standard probability format, with explicit prior probabilities and likelihoods. The standard probability format is not the most meaningful to subjects; frequency formats better facilitate Bayesian reasoning (Cosmides & Tooby Reference Cosmides and Tooby1996; Gigerenzer & Hoffrage Reference Gigerenzer and Hoffrage1995). Personal experience of environmental probabilities may be even more effective. When environmental probabilities are learned through personal experience, the vast majority of subjects maximize the probability of a correct guess (probability gain), rather than impact or information gain (Nelson et al., submitted). Note that impact (which the feature difference strategy implements) more reliably approximates probability gain than do Bayesian diagnosticity or log diagnosticity (Nelson Reference Nelson2005), and impact is easily calculated when the standard probability format is used.

Is cognition optimal? Adaptation can be impressive. Insects' flight length distributions appear well-calibrated to natural environments (Viswanathan et al. Reference Viswanathan, Buldyrev, Havlin, da Luz, Raposo and Stanley1999). But the modern world is evolutionarily novel. For instance, sugar, fat, and salt are available in unprecedented abundance. Similarly, modern media may exaggerate the incidence of plane crashes versus car crashes, or terrorism versus heart disease. The increasing rate of human genetic evolution (Hawks et al. Reference Hawks, Wang, Cochran, Harpending and Moyzis2007) may facilitate adaptation to some modern environments, over phylogenetic time.

Among topics of interest in Bayesian and rational analysis, such as perception (e.g., Hoffman, in press), memory, information search, and category formation, the correct function to optimize is seldom clear. Baron (Reference Baron, Koehler and Harvey2004) noted that utilities are formed on the basis of reflection, and are constantly being modified. As a pragmatic matter, cognitive science would be wise to treat candidate normative models in similar fashion (also see McKenzie Reference McKenzie2003). When there are clear and robust discrepancies between human behavior and a particular theoretical model, the normative status of the theoretical model should be reconsidered, as well as the rationality or adaptiveness of the human behavior.