Elqayam & Evans (E&E) argue that normative theories are problematic and have impeded theoretical development in the psychology of thinking and reasoning. In this commentary, I articulate additional arguments for the assertion that using normative correctness as a diagnostic tool has impeded our ability to formulate good cognitive theories of the processes that generate inferences. For this, I have assumed traditional models of probability and logic, although the comments are germane to any normative standard used to evaluate performance.
One of the primary goals of a normative theory is to serve as a standard that specifies how people should reason and to then evaluate how well they achieve this standard. Reliance on normativism in thinking and reasoning research was evident from the first papers published on this subject (e.g., Wilkins Reference Wilkins1928), and evolved into a long-standing effort to explain why judgements and inferences often defied normative prescriptions (e.g., Henle Reference Henle1962; Kahneman & Tversky Reference Kahneman and Tversky1973) that remains to this day.
In fact, as a theoretical and empirical standard for cognitive science, the proscriptions of normativism fail us badly. As cognitive psychologists, our goal is to describe how situations, problems, and goals are represented; how knowledge is recruited to the problem space; the mechanisms by which inferences are computed; and how attentional processes and capacity limitations constrain these operations. Reliance on normative standards diminishes our ability to achieve these goals in numerous ways, not least of which is that it perpetuates reliance on an impoverished form of data, namely, a binary outcome variable. Thus, complex inferential processes that access a wide array of complex cognitive functions are reduced to a binary classification of either normative or non-normative as the (often sole) dependent measure. Alternative measures, such as reporting whether an inference has been accepted, whether a card on the Wason task has been selected, or whether a probability estimate is “close to a base rate,” at least preserve some element of the actual decision: its task-relevant outcome. However, like the normative measure, this single piece of information tells us little about the processes that produced it. As a discipline, the field of thinking and reasoning has been slow to incorporate other processing measures, such as eye tracking, reaction times, confidence measures, and so on, which would provide a much richer view of complex processes than can be gleaned from a single binary measure.
So impoverished is a normative index as an outcome measure that it can be flatly misleading. Indeed, it is easily demonstrable that the same processes may underlie both normative and non-normative responses. For example, on conditional reasoning tasks, both logically valid and invalid responses can be achieved by the same process, namely, the recruitment of counter examples to the conditional (Thompson Reference Thompson1994). Conversely, a normative (or non-normative) response may be achieved by a variety of processes. A normatively incorrect response may arise because a salient heuristic process produced a prepotent answer; because the reasoner did not understand the instructions; because the reasoner tried but failed to compute a normatively correct answer; and so on. Similarly, a normatively correct response may be achieved because the reasoner knows and applies the relevant normative standard; because the reasoner guessed; because he or she used a strategy, such as the recognition heuristic (Goldstein & Gigerenzer Reference Goldstein, Gigerenzer, Gigerenzer and Todd1999) that often produces correct answers; and so forth.
As an argument in favour of the utility of normative theories, one might point to the evidence demonstrating that persons of higher cognitive ability are able to suppress a tendency to give a “heuristic” response and produce normatively correct answers on reasoning tasks (Stanovich Reference Stanovich1999). This research implies that there is value in the normative label, in that it embodies a quality that differentiates between high and low capacity reasoners. Again, however, having such a limited measure of reasoning performance means that these correlations between capacity and normative responses, in and of themselves, tell us little about how such suppression is achieved and may, in fact, tell us more about the reasoner's capacity to understand and encode instructions than about fundamental processes of reasoning. In support of this argument, consider that many concepts such as logical necessity are likely new to most reasoners, such that applying them consistently to a set of novel reasoning problems will certainly require capacity. In cases where the task instructions are less onerous, correlations with cognitive capacity may arise for any number of reasons, ranging from having access to appropriate normative rules, to recognizing their relevance in the current context, and having the motivation to do so (Stanovich Reference Stanovich, Evans and Frankish2009a).
Further evidence that normative versus non-normative responses tell us little about the mechanisms that produced them comes from a series of studies recently conducted in my lab. These studies employ a paradigm developed to study processes that monitor inferences and determine when additional analytic thinking is warranted. In this paradigm, participants are asked to give an initial, intuitive answer to a problem, make a metacognitive judgement about how right that answer feels, and are then allowed to take as much time as needed to give a final answer. The metacognitive experience that accompanies an initial decision predicts a lot about the effort that people put into solving a problem. Specifically, a strong feeling of rightness about a decision determines the amount of time spent thinking about a problem and whether or not the initial answer is changed in favour of another. Importantly, a strong initial feeling of rightness does not reliably predict whether the final answer is normatively accurate, despite the fact that it exerts substantial control over subsequent analytic reasoning. Moreover, in several studies, we have observed that when people do change their answers, they are often just as likely to change from a normatively correct answer to a wrong one, as vice versa (Shynkaruk & Thompson Reference Shynkarkuk and Thompson2006; Thompson et al., under review). Understanding what motivates an answer change, what constraints an answer must satisfy to be retained, the information that is recruited to rework the answer, and so on, will tell us a lot about human reasoning. Knowing that the final outcome is normative tells us virtually nothing about the underlying mechanisms.
Elqayam & Evans (E&E) argue that normative theories are problematic and have impeded theoretical development in the psychology of thinking and reasoning. In this commentary, I articulate additional arguments for the assertion that using normative correctness as a diagnostic tool has impeded our ability to formulate good cognitive theories of the processes that generate inferences. For this, I have assumed traditional models of probability and logic, although the comments are germane to any normative standard used to evaluate performance.
One of the primary goals of a normative theory is to serve as a standard that specifies how people should reason and to then evaluate how well they achieve this standard. Reliance on normativism in thinking and reasoning research was evident from the first papers published on this subject (e.g., Wilkins Reference Wilkins1928), and evolved into a long-standing effort to explain why judgements and inferences often defied normative prescriptions (e.g., Henle Reference Henle1962; Kahneman & Tversky Reference Kahneman and Tversky1973) that remains to this day.
In fact, as a theoretical and empirical standard for cognitive science, the proscriptions of normativism fail us badly. As cognitive psychologists, our goal is to describe how situations, problems, and goals are represented; how knowledge is recruited to the problem space; the mechanisms by which inferences are computed; and how attentional processes and capacity limitations constrain these operations. Reliance on normative standards diminishes our ability to achieve these goals in numerous ways, not least of which is that it perpetuates reliance on an impoverished form of data, namely, a binary outcome variable. Thus, complex inferential processes that access a wide array of complex cognitive functions are reduced to a binary classification of either normative or non-normative as the (often sole) dependent measure. Alternative measures, such as reporting whether an inference has been accepted, whether a card on the Wason task has been selected, or whether a probability estimate is “close to a base rate,” at least preserve some element of the actual decision: its task-relevant outcome. However, like the normative measure, this single piece of information tells us little about the processes that produced it. As a discipline, the field of thinking and reasoning has been slow to incorporate other processing measures, such as eye tracking, reaction times, confidence measures, and so on, which would provide a much richer view of complex processes than can be gleaned from a single binary measure.
So impoverished is a normative index as an outcome measure that it can be flatly misleading. Indeed, it is easily demonstrable that the same processes may underlie both normative and non-normative responses. For example, on conditional reasoning tasks, both logically valid and invalid responses can be achieved by the same process, namely, the recruitment of counter examples to the conditional (Thompson Reference Thompson1994). Conversely, a normative (or non-normative) response may be achieved by a variety of processes. A normatively incorrect response may arise because a salient heuristic process produced a prepotent answer; because the reasoner did not understand the instructions; because the reasoner tried but failed to compute a normatively correct answer; and so on. Similarly, a normatively correct response may be achieved because the reasoner knows and applies the relevant normative standard; because the reasoner guessed; because he or she used a strategy, such as the recognition heuristic (Goldstein & Gigerenzer Reference Goldstein, Gigerenzer, Gigerenzer and Todd1999) that often produces correct answers; and so forth.
As an argument in favour of the utility of normative theories, one might point to the evidence demonstrating that persons of higher cognitive ability are able to suppress a tendency to give a “heuristic” response and produce normatively correct answers on reasoning tasks (Stanovich Reference Stanovich1999). This research implies that there is value in the normative label, in that it embodies a quality that differentiates between high and low capacity reasoners. Again, however, having such a limited measure of reasoning performance means that these correlations between capacity and normative responses, in and of themselves, tell us little about how such suppression is achieved and may, in fact, tell us more about the reasoner's capacity to understand and encode instructions than about fundamental processes of reasoning. In support of this argument, consider that many concepts such as logical necessity are likely new to most reasoners, such that applying them consistently to a set of novel reasoning problems will certainly require capacity. In cases where the task instructions are less onerous, correlations with cognitive capacity may arise for any number of reasons, ranging from having access to appropriate normative rules, to recognizing their relevance in the current context, and having the motivation to do so (Stanovich Reference Stanovich, Evans and Frankish2009a).
Further evidence that normative versus non-normative responses tell us little about the mechanisms that produced them comes from a series of studies recently conducted in my lab. These studies employ a paradigm developed to study processes that monitor inferences and determine when additional analytic thinking is warranted. In this paradigm, participants are asked to give an initial, intuitive answer to a problem, make a metacognitive judgement about how right that answer feels, and are then allowed to take as much time as needed to give a final answer. The metacognitive experience that accompanies an initial decision predicts a lot about the effort that people put into solving a problem. Specifically, a strong feeling of rightness about a decision determines the amount of time spent thinking about a problem and whether or not the initial answer is changed in favour of another. Importantly, a strong initial feeling of rightness does not reliably predict whether the final answer is normatively accurate, despite the fact that it exerts substantial control over subsequent analytic reasoning. Moreover, in several studies, we have observed that when people do change their answers, they are often just as likely to change from a normatively correct answer to a wrong one, as vice versa (Shynkaruk & Thompson Reference Shynkarkuk and Thompson2006; Thompson et al., under review). Understanding what motivates an answer change, what constraints an answer must satisfy to be retained, the information that is recruited to rework the answer, and so on, will tell us a lot about human reasoning. Knowing that the final outcome is normative tells us virtually nothing about the underlying mechanisms.