R2.1. Rule-based versus associative ≠ normative versus counter-normative

Evans & Elqayam, Gaissmaier, Straubinger, & Funder (Gaissmaier et al.), and Lagnado & Shanks surprised us by attributing to us a claim that we did not make. We never did and never would align the dual processes of associative and rule-based reasoning with the normative versus non-normative distinction. Indeed, Sloman (Reference Sloman1996a) explicitly denies such a parallel and points out that normative rules are only one kind of rule used by the rule-based system. Of course rules can lead to errors and of course associations frequently lead to correct responses; after all, people mostly do pretty well at achieving their goals. The only claim we made in the target article is that base-rate neglect can be remedied when elementary rules of set theory are applied. This is hardly a broad claim about how error prone each system is.

R2.2. On the relation between dual processes and nested sets

Of course, whether or not there are two systems remains an open question and, as Keren, van Rooij, & Schul (Keren et al.), Mandel, Samuels, and Trafimow point out, the claim is conceptually independent of the nested sets hypothesis. Nonetheless, the dual process hypothesis remains the simplest viable framework for motivating the nested sets hypothesis for several reasons.

First, dual process theory offers a general framework providing background assumptions to explain the variety of phenomena addressed by the nested sets hypothesis (see Table 2 of the target article). “Inaccurate frequency judgments,” for example, result primarily from associative processes (see sect. 2.6 of the target article and Fantino & Stolarz-Fantino), whereas the facilitative role of set representations in deductive inference depends primarily on rule-based processes (see sect. 2.9 of the target article). We know of no account of the variety of predictions summarized in Table 2 that does not assume more than one cognitive system. The associative versus rule-based distinction has the advantage of providing a common account of these diverse phenomena and has proven useful for interpreting a variety of judgment effects (Kahneman & Fredericks Reference Kahneman, Frederick, Gilovich, Griffin and Kahneman2002), especially probability judgment such as the conjunction fallacy (Tversky & Kahneman Reference Tversky, Kahneman, Kahneman, Slovic and Tversky1982b; Reference Tversky and Kahneman1983; cf. Sloman Reference Sloman1996b).

Second, in the absence of systematic studies that assess the role of associative and rule-based processes in probability judgment (a point made by De Neys), it can be argued that the facilitative effect of nested set representations on Bayesian inference results from (1) different rule-;[?tjl]>based processes, a possibility raised by Gigerenzer & Hoffrage, Keren et al., and Mandel, or from (2) multiple associative processes, as Lagnado & Shanks argue. These proposals represent logical possibilities but, in the broad form in which they are stated, they have little empirical content. Consider, for example, Lagnado & Shanks's proposal that one associative system results in non-Bayesian responses, whereas another associative system is engaged when people “see” the set inclusion relations illustrated by Euler diagrams and draw the Bayesian solution. Lagnado & Shanks do not specify the associative processes that give rise to normative versus non-normative responses. How do associative processes implement the elementary set operations or whatever operations are responsible for Bayesian responding? We suspect that if the proposal were spelled out, they would end up with a dual process theory that includes associative and rule-based operations.

R2.3. Why rule-based and associative?

Various versions of a two-systems hypothesis have been offered. Our claim (in contrast to Brainerd, Evans & Elqayam, Reyna & Mills, and Wolfe) is that Sloman's (Reference Sloman1996a) characterization of the associative system is consistent with cases of base-rate neglect (people rely on associations based on statistical regularities embodied by events in experience), and that his characterization of rule-based reasoning is consistent with reasoning during nested sets facilitation (deliberative reasoning about set relations based on rules of combination).

In support of our position, Evans & Elqayam point out that there is an association (the one asserted by Kahneman & Tversky [Reference Kahneman and Frederick1973] in their original demonstration of the phenomenon) that explains the majority response, namely the association between the hypothesis being evaluated (the presence of breast cancer) and the case data that provide diagnostic information about it (the probability of a positive mammogram given breast cancer). Indeed, this observation supports our claim that associative operations often lead to base-rate neglect. In the context of the Medical Diagnosis problem, this occurs when judgments are based on the association between a positive mammogram and breast cancer, or, in Kahneman and Tversky's terms, when judgments reflect how representative a positive mammogram is of breast cancer (see sects. 1.2.5 and 2.3 of the target article).

Of course not all responses that neglect base rates are associative in this sense (as Gigerenzer & Hoffrage show convincingly). Prior to their assertion that all reasoning on this task is associative, Lagnado & Shanks point out that one response that is observed involves a rule: Sometimes people report the complement of the false-alarm rate. That is true. This response involves a subtraction from 1. People find the Medical Diagnosis problem and related problems very difficult, and use a host of different strategies in their struggle to generate a reasonable answer. Many of those strategies involve rules. However, the appeal to a representative outcome does partially account for their response. We actually agree with most of Evans & Elqayam and Macchi & Bagassi's description of what goes on when people try to solve the Medical Diagnosis problem. These commentators suggest, however, that the process they describe implies that errors are produced by a pragmatic system. We cannot see what explanatory purchase this provides. All systems of reasoning must be sensitive to pragmatics in order for their output to be relevant to reasoners' goals and concerns.

An alternative dual process theory is advocated by Griffin et al., who propose that “The conditions under which … [nested set representations] promote base-rate use may be more parsimoniously organized under the broader notion of case-based judgment.” We find Griffin et al.'s proposal intriguing but difficult to assess in the absence of detail concerning the cognitive operations that give rise to the “strength of impression of the case at hand,” or in the absence of a proposal about how this construct is measured and differentiated from alternative accounts. The case-based theory does appear to be inconsistent with the large body of evidence we review demonstrating Bayesian inference facilitation by virtue of employing samples of category instances that would not seem to strengthen single-case impressions (see sect. 2 of the target article).

Griffin et al. suggest that all forms of base-rate facilitation can be explained in terms of single-case impressions. For instance, they argue that judgments drawn from Euler diagrams depend on case-specific information (see sect. 2.5 of the target article). According to their view, “Diagrams prompting an immediate comparison of the size of circles may allow a low-level perceptual computation to solve the problem.” We suspect that facilitation by nested sets takes advantage of visual representations that allow us to see in the world, or in our mind's eye, the relation between relevant sets. But the nested sets hypothesis requires a number of additional steps involving symbol manipulation in order to apply this representation to solving a base-rate problem. First, each set must be labeled; second (as Patterson points out), the correct sets must be chosen; and third, a symbolic response (a number) must be generated. Thus, even in the context of diagrammatic representations, Bayesian inference cannot be reduced to “a low-level perceptual computation,” without appealing to symbolic operations. Whatever the right theory may be to explain base-rate neglect and respect, these forms of symbol manipulation require an account and Griffin et al.'s proposal does not currently offer one. The case-based theory may explain some instances of facilitation that are outside the scope of the nested sets hypothesis, but it does not seem to be a substitute for it.

Brainerd and Reyna & Mills review evidence that supports a dual process theory of judgment, and, in the process, cover some of the history of the ideas that we neglected. These commentators offer the denominator neglect model of inductive inference, a special case of fuzzy trace theory, as an account of base-rate neglect. According to this view, errors in probabilistic inference result from the failure to represent and attend to all of the information present in a nested set relation, specifically the information captured by the denominator of a Bayesian ratio. While we obviously agree with the claim about nested sets, we are less comfortable associating what is and what is not neglected with terms of a mathematical expression. As we do not believe that the mind embodies a mental analogue of Bayes' theorem (see sect. 2.8 of the target article), we also do not believe that judgment errors correspond to neglect of terms of the theorem. Rather, we believe that, in cases of base-rate neglect, people are doing something other than trying to map statistical information from the problem onto a mathematical equation. Specifically, we believe that errors result from a failure to map the problem onto a mental representation of the conceptual relations among sets. According to the nested set hypothesis, representing conceptual relations among sets affords a natural mapping to a correct numerical response. In the case of rule-based processing, this requires several forms of symbol manipulation (e.g., combination rules) that operate from a qualitative representation of structural relations among sets (see sect. 1.2.5 of the target article).

Reyna & Mills distinguish their fuzzy-trace theory dual process model from our dual process account by stating that the former predicts that normative judgment results from associative processes (System 1 operations), whereas counter-normative judgment results from a focus on quantitative details (System 2 operations). This prediction appears to be inconsistent with the large body of evidence demonstrating that (a) under the right conditions, heuristics can produce systematic errors in judgment (for a recent review, see Gilovich et al. Reference Gilovich, Griffin and Kahneman2002), and (b) Bayesian facilitation is sometimes a result of deliberative, rule-based operations (see sect. 2 of the target article).

R2.4. Evidence in favor of dual process theory

Some of the commentaries provide strong arguments in support of our perspective. Fantino & Stolarz-Fantino show that base-rate neglect is well-captured by associative principles in the context of trial-by-trial presentations (they also provide additional support for our claim that natural frequency formats are not sufficient to eliminate base-rate neglect). On the flip side, Over shows that representation via nested sets is equivalent to the logic of the excluded middle. Taken together, these observations suggest that very different inferential principles apply to (at least some) cases of base-rate neglect and to cases of facilitation via nested set representations. The fact that different inferential principles apply does not entail that different systems of representation and processing apply, but the dual systems hypothesis does offer a simple explanation that is consistent with this and a host of other data (cf. Evans Reference Evans2003).

Newell & Hayes, who offer several objections to our proposal, also point to results that favor our perspective. We agree that much can be learned from assessments of base-rate usage in the category-learning domain. Those data are well explained by an associative theory that takes account of differential attention to features (Kruschke & Johansen Reference Kruschke and Johansen1999). That is one reason why we appeal to associative processes to explain performance in the absence of additional structural cues. As Newell & Hayes point out, there is nothing in studies of category learning that corresponds to making nested sets transparent. Of course, only if there were would the need arise to invoke the rules of nested set representations.

Patterson points to the generality of nested set representations and their potential role in deductive, modal, deontic, and causal reasoning. His proposal shares with Johnson-Laird's (Reference Johnson-Laird1983) domain-general theory that these forms of inference are represented in terms of sets of possibilities. The prediction that nested set representations will facilitate deductive inference is supported by evidence reviewed in the target article (see sect. 2.9). Patterson's suggestion of assessing the descriptive validity of the Leibnizean principle (If all As are Bs, then everything that is related in manner R to an A is related in manner R to a B) has already been explored in the context of category induction (Sloman Reference Sloman1998). This research demonstrates that the Leibnizean principle is obeyed only when the category relation is made explicit – further implicating the role of nested set representations in reasoning. Although we agree with Patterson that representing subset relations can facilitate probability judgment and deductive reasoning, we are not optimistic that the nested sets theory will support a general framework for representing modal, deontic, and causal relations (Sloman Reference Sloman2005).

Butterworth and Sun & Wang review evidence addressing the cognitive and neural foundations of numeric processing. Sun & Wang provide evidence that the mind embodies a coarse number scale consisting of qualitative categories. The reviewed neuroimaging evidence demonstrates that exact calculations recruit the language system, whereas approximate calculations rely on visuo-spatial representations of numbers mediated by parietal areas. Butterworth suggests that the latter reflects a “classic Fodorian cognitive module,” whereas Sun & Wang argue that together these systems may provide the neural foundations for the proposed dual systems theory (cf., Goel Reference Goel, Holyoak and Morrison2005). We agree that intuitive probability judgment depends on qualitative representations and find the cited neuroimaging evidence suggestive (for a recent review of the neuroscience literature on reasoning, see Barbey & Barsalou, in press).

Schurr & Erev and Thomas address the degree to which the reviewed findings generalize to real-world settings. Schurr & Erev raise an important distinction between decisions from description versus experience. In contrast to the underutilization of base-rates observed in decisions from description (see sects. 1 and 2 of the target article), Schurr & Erev argue that decisions from experience result in base-rate overweighting. Although Schurr & Erev make a convincing case for their proposal, we are not convinced that the cited example involves representing structural relations among sets. It seems rather to involve making alternative interpretations of a stimulus more available. It would be analogous, in the Medical Diagnosis problem, to suggesting that the positive result has a different interpretation. Although Schurr & Erev's proposal may not directly inform the nested sets hypothesis, the distinction they raise is certainly of value in its own right.

Uhlmann et al. and Whitney et al. offer important insights that extend the proposed dual process theory to include social-psychological and emotional factors. We appreciate Uhlmann et al.'s suggestion that motivations can moderate whether people rely on rule-based versus associative processes and believe that these factors should be incorporated into any complete theory of judgment. Whitney et al.'s proposal that the dual process theory can be integrated with the literature on affective influences on reasoning offers a worthwhile theoretical challenge.

R3.1. Clarifying the intent and value of Gigerenzer and Hoffrage (Reference Gigerenzer and Hoffrage1995)

As Kleiter makes crystal clear, our intent was to argue that facilitation on base-rate problems often results from clarifying the structural relations among the relevant sets referred to in the problem. Gigerenzer & Hoffrage and Brase seem to concur, suggesting (happily) that there is wide agreement on this issue. Indeed, Gigerenzer & Hoffrage and Barton, Mousavi, & Stevens (Barton et al.) argue that this was always the intended meaning of Gigerenzer and Hoffrage (Reference Gigerenzer and Hoffrage1995) and that they have been repeatedly misunderstood as having suggested that frequencies of any type arising through natural sampling are sufficient for facilitation. In fact, we were very careful to distinguish normalized from non-normalized frequencies, but we (like many others) believed that Gigerenzer and Hoffrage (Reference Gigerenzer and Hoffrage1995) were trying to say something other than that there are computational advantages to what we have here described as the nested sets hypothesis (Tversky & Kahneman Reference Tversky and Kahneman1983).

On reading Gigerenzer & Hoffrage, we find it intriguing that so many researchers are guilty of the identical apparent misinterpretation of Gigerenzer and Hoffrage (Reference Gigerenzer and Hoffrage1995). It might have to do with passages like the following one from Gigerenzer and Hoffrage (Reference Gigerenzer and Hoffrage1995).

They also refer to the natural frequency hypothesis as “our evolutionary argument that cognitive algorithms were designed for frequency information acquired through natural sampling” (Gigerenzer & Hoffrage Reference Gigerenzer and Hoffrage1995, p. 699). Further quotes from that paper appear in our target article.

Gigerenzer & Hoffrage point out that the evolutionary argument has nothing to do with deriving predictions from the natural frequency hypothesis and here we agree. But it does not seem unreasonable to infer from their own language that these authors put scientific weight on the claim that there exists an evolved frequency-sensitive algorithm. Of course, our review also makes clear – pace Brase – that Gigerenzer & Hoffrage's own theorizing has not been entirely consistent over the years (admittedly, neither has ours).

Nevertheless, we do not entirely agree that Gigerenzer & Hoffrage's most recent proposal completely converges with the nested sets hypothesis. According to Gigerenzer & Hoffrage, “the question is how and why reasoning depends on the external representation of information.” We believe that the critical question is: How and why does reasoning depend on the internal representation of information? Our hypothesis concerns mental representations. The natural frequency hypothesis, even in its new form, is “about the general question of how various external representations facilitate Bayesian computations” (Gigerenzer & Hoffrage's commentary). But the findings we review suggest to us that different external representations (e.g., natural frequencies, chances) map onto the same internal representation.

More specifically, Gigerenzer & Hoffrage's theory is that different textual formats map onto different equations. We don't believe that the mind is composed of equations even in the form of algorithms. Rather, we believe that people invoke different combination rules in a highly context-specific way that depends on techniques they have learned or figured out themselves. The critical mapping process is not from text to mathematical equation, but rather, in the case of rule-based processing, from text to a qualitative representation of structural relations among sets.

Nested set structures do not simplify Bayesian computations themselves; rather they suggest a cognitive representation that affords simple computations. As a result, the nested sets hypothesis cannot be reduced to the equations cited in Gigerenzer & Hoffrage's commentary. Furthermore, the additional predictions cited in their commentary do not bear on the reviewed findings as they suggest. Predictions 2, 3, and 4 are not addressed in our review because they do not distinguish between competing theoretical accounts – nor do they directly bear on the target article's assessment of Gigerenzer and Hoffrage's (Reference Gigerenzer and Hoffrage1995) “evolutionary argument that cognitive algorithms were designed for frequency information” (p. 699).

R3.2. Does natural sampling support the natural frequency hypothesis?

Natural frequency theorists motivate the evolutionary argument that the mind is designed to process natural frequencies by appealing to the natural sampling framework (Kleiter Reference Kleiter, Fischer and Laming1994). As Kleiter makes clear in his commentary, however, the natural sampling framework is based on a statistical model that is not consistent with the psychological theory advocated by natural frequency theorists. In particular, the natural sampling framework depends on several assumptions that are rarely satisfied in the natural environment, including complete data, “additive frequencies in hierarchical tree-like sample/subsample structure,” and random sampling (see also Kleiter Reference Kleiter, Fischer and Laming1994 and sect. 3.1 of the target article). Natural frequency theories that appeal to sequential sampling and evolutionary plausibility have little to do with natural sampling in Kleiter's original sense. Kleiter points out in the commentary that the assumptions of his framework are rarely satisfied in the natural environment and, as a result, the computational advantage of natural sampling has nothing to do with ecological validity.

R3.3. What are natural frequencies?

Barton et al., Brase, and Gigerenzer & Hoffrage argue that the simple frequencies employed by Brase (Reference Brase2002b)Footnote ¹ do not represent natural frequencies. These commentators say that single numerical statements (e.g., 1 out of 2) are simple frequencies, whereas natural frequencies necessarily represent the structural relations among the operative sets, or, in their language, the structure of the entire tree diagram. This view is inconsistent with the description of natural frequencies in recent work such as that of Zhu and Gigerenzer (Reference Zhu and Gigerenzer2006), in which the authors talk of “natural frequencies such as 1 out of 2” (p. 15). Moreover, for binary events single numerical statements can satisfy the definition of natural frequencies. Consider, for example, the single numerical statement “I win poker 1 out of 10 nights.” This statement directly implies that “I lose poker on 9 out of 10 nights” and therefore represents the size of the reference class (e.g., “10 nights total”), in addition to the relevant subset relations (e.g., “1 night I win” and “9 nights I lose”), and thus the structure of the event tree.

The clarification offered by Barton et al., Brase, and Gigerenzer & Hoffrage is helpful of course, for it indicates that the natural frequency theory, like nested sets, concerns the representation of the structural relations among the events. Both positions leave open questions about the conditions of base-rate neglect and respect: When should we expect judgments to be veridical? How much can people represent and what are the computational demands of different problems? We do not believe that ecological considerations or appeals to problem formats provide an answer to these questions. These questions require an analysis of internal mental representations, their power and demands, as well as the conditions that elicit them.

Brase raises a further objection concerning the conclusions drawn from Brase et al. (Reference Brase, Fiddick and Harries2006). He argues that we “try to infer cognitive abilities and structures from data showing that incentives affect performance.” In fact, our conclusion about domain general cognitive processes does not depend on the findings Brase mentions concerning monetary incentives. Our claim is that “a higher proportion of Bayesian responses is observed in experiments that […] select participants with a higher level of general intelligence … [which is] consistent with the view that Bayesian reasoning depends on domain general cognitive processes to the degree that intelligence is domain general” (target article, sect. 2.2, para. 5).

R3.4. Are natural frequencies and chances equivalent representations?

By definition, chances refer to the likelihood of a single event (see Barton et al.) determined by a distribution of possibilities, not a sample of observations. Consequently, as we have pointed out in Note 4 of the target article, chances are not obtained by “counting occurrences of events as they are encountered and storing the resulting knowledge base for possible use later” (i.e., natural sampling; Brase Reference Brase2002b, p. 384). In this sense, chances are therefore distinct from natural frequencies. Yet Gigerenzer & Hoffrage and Brase propose that chances are equivalent to natural frequencies and, as a consequence, that the natural frequency hypothesis predicts that chance representations will facilitate Bayesian inference. We all seem to agree on what facilitates Bayesian inference, but broadening the definition of natural frequencies to include chances appears to undermine the claim that “cognitive algorithms were designed for frequency information, acquired through natural sampling” (Gigerenzer & Hoffrage Reference Gigerenzer and Hoffrage1995, p. 686; see target article sect. 1.2.2, para. 3).

We welcome the recent articulation of the natural frequency hypothesis and believe that the current formulation is a roughly accurate characterization of some of the conditions that lead to base-rate respect. By broadening the definition of natural frequencies to include chances, Gigerenzer & Hoffrage's proposal implies that (1) cognitive algorithms were not designed over the course of human evolution to process natural frequencies rather than the likelihood or chance of a single event, (2) the theory no longer appeals to the natural sampling framework (which cannot encode chances), and, finally, (3) the findings are not motivated by the ecological rationality program, which claims that our current environment represents statistical information in the form of natural frequencies “as actually experienced in a series of events” rather than conveying the likelihood or chance of a single event (Gigerenzer & Hoffrage, Reference Gigerenzer and Hoffrage1995, p. 686). With these clarifications, we agree with Gigerenzer & Hoffrage and Brase that the natural frequency and nested sets hypotheses are hard to distinguish. Of course, as the key ideas have to do with structural relations among sets and not with frequency counts, we believe the term “nested sets” is more adequate descriptively.