We often think of multidisciplinary work as being useful because it can expand our discipline and can lead to challenges to pervasive biases that we may not be able to identify within our own discipline. In this paper, I examine work on human reasoning as a case study to show some of the pitfalls of multidisciplinary work. First, I give an historical overview of the study of human reasoning by experimental psychologists, some of the studies used to examine reasoning in natural languages, and the conclusions drawn from these studies. Second, I outline some experimental results that challenge these interpretations. The history of interdisciplinary work in studies of human reasoning is particularly interesting because I think it shows ways that interdisciplinary research can solidify rather than challenge pervasive preconceptions in a particular field, and can narrow rather than expand the boundaries of a particular discipline. Specifically, I will argue that work on reasoning done by experimental psychologists in the past 50 years could be taken as a test of particular norms of reasoning that many philosophers take as certain, unquestionable, and non-contextual. Work in experimental psychology has shown that these normative systems fail to model reasoning in natural language contexts (primarily that of lay reasoners but also by those we would consider expert reasoners.) In the past, philosophers have taken this data to imply that natural language users are simply poor reasoners and that, therefore, untutored intuitions about logical terms can be ignored. I will argue that philosophers could instead take this psychological research as motivation to amend these normative models or radically change their ideas about how these models are applied to reasoning in natural languages. This psychological data has the potential to lead to two possible conclusions—either natural language users are mistaken in widespread ways about the meaning of the logical terms that they regularly use, or that our normative systems for understanding logical terms fail to ‘match-up’ to natural language in some fairly concerning ways. In this particular example, the interdisciplinary work done by philosophers and psychologists rejected the conclusion that would challenge the pervasive biases (about what constitutes good reasoning) within the field. I argue that interdisciplinarity, in this case, failed to broaden the borders of the discipline and reinforced some of the biases within philosophy. I conclude by offering some tentative suggestions about what led to the failure of interdisciplinarity in this particular case.
When experimental psychologists first started to study human reasoning, they designed their experiments around norms of reasoning that were adopted from philosophers working in logic. For example, tasks designed to study deductive reasoning relied upon some of the basic inference forms from propositional logic and classical (Aristotelian) inference forms as outlined in introduction to logic textbooks.Footnote 1 Tasks studying statistical reasoning relied upon rules taken from Thomas Bayes and Andrey Kolmogarov. The primary lesson learned through these initial studies was that subjects tend not to reason in accordance with these particular norms. I will outline some deductive and inductive reasoning tasks used in the study of reasoning and describe the types of responses typically seen in these tasks. Most subjects failed to give the expected answers (the answer that the normative systems would proscribe), from which most experimental psychologists concluded that subjects did not use these particular norms while reasoning. They failed, in short, as descriptions or predictors of human reasoning.
One of the most ubiquitous types of study used to examine deductive reasoning is the Wason selection task. In the Wason selection task subjects are given four cards with A, C, 3, and 7 on each and asked which cards they need to turn over to test the conditional statement ‘if a card has an A on one side, it will have a 7 on the other.’ Researchers initially expected subjects to give answers that would correspond to a material conditional interpretation. In other words, subjects were expected to select the falsifying cards of the antecedent, A and the false consequent, 3. Subjects overwhelmingly selected the A and 7 cards to turn over and not the A and 3 cards that the normative account of the material conditional tells us have the ability to falsify this conditional. This led researchers to conclude that subjects either have a faulty truth table for the material conditional or that they fail to reason properly on this task. These surprising results from the Wason selection task experiments invited researchers to explore how actual reasoners reason, including by examining what are known as content effects, which are extra-logical effects that impact reasoning in natural languages. Now the Wason selection task is used primarily to study these content effects, i.e., to determine which changes in the set-up or wording of the task impacts subject responses. One of the effects seen in the Wason selection task (and in the truth table evaluation taskFootnote 2) is called confirmation bias. In the Wason selection task, less than 10% of peopleFootnote 3 opt to turn over the p and not-q cards (when the conditional they are given is ‘if p, then q’) which have the potential to falsify the conditional (given the standard truth-functional account of conditionals). Instead, subjects more frequently opt to turn over the q card and the p card. Peter Wason and Philip Johnson-Laird conclude, “typical results were: p and q cards, 46 percent; p card only, 33 percent; p, q, and not-q cards, 7 percent; and p and not-q cards, 4 percent.”Footnote 4 These results led to a description of confirmation bias (in Wason’s early papers, it is called verification bias). In an early study done by Wason, subjects were found to select most often the true consequent and true antecedent and rarely to select the falsifying case of false consequent. This led Wason to propose the theory that subjects are operating with a defective truth table, i.e., a truth table that conflicts with the standard truth functional account. Another bias seen in the Wason selection task is called matching bias. Matching bias refers to the tendency to select values for cards that match the rule. For example, when subjects are given the conditional ‘If p, then not q,’ they tend to choose to examine the potentially falsifying cases of the true antecedent and false consequent more frequently than they do when they are given a conditional with a positive antecedent and consequent. In the Wason selection task, subjects choose to turn over the card that will verify the antecedent more often on rules with an affirmative antecedent and cards that would falsify the antecedent more often when the antecedent is negated in the rule. Likewise, subjects choose to turn over the card that will verify the consequent more often when the consequent is affirmed and attempt to falsify the consequent more often when the consequent is negated.Footnote 5 In 1972, Jonathan Evans performed a study where subjects were given a conditional rule and a possible truth table and asked which line on the truth table falsifies the conditional rule given. He noted that subjects choose the line where the antecedent is true and the consequent false as the correct falsifier more often when the rule given is ‘if p, then not q’ and least often when the rule given is “if not p, then q.”Footnote 6 One early attempt to explain matching bias was to argue that subjects are employing an heuristic, i.e., subjects turn over the p and q cards simply because they have just seen a p and a q. This assumption does little to explain subjects’ selections—it merely discounts them as extra-logical errors.
We also see a failure of normative models when testing inductive reasoning, with subjects committing fallacies, such as the gambler’s fallacy (belief that non-causally connected previous events will have an impact on the probability of a future event), the base-rate fallacy (where people ignore general statistical data that is relevant in a particular case and focus only on the data that pertains to the specific case), and the conjunction fallacy (where people rate the likelihood of a conjunction of properties as higher than either one of the conjuncts alone). In tasks related to statistical reasoning, subjects tend to ignore base rates, attend to statistically irrelevant information, and violate some of Kolmogorov’s axioms of probability theory (e.g., as seen in the conjunction fallacy).
For example, in a series of experiments Daniel Kahneman and Amos TverskyFootnote 7 showed that subjects can either undervalue or ignore relevant statistical information given in a task. In these experiments, subjects were given the following problem:
A panel of psychologists have interviewed and administered personality tests to 30 engineers and 70 lawyers, all successful in their respective fields. On the basis of this information, thumbnail descriptions of the 30 engineers and 70 lawyers have been written. You will find on your forms five descriptions, chosen at random from the 100 available descriptions. For each description, please indicate your probability that the person described is an engineer, on a scale from 0 to 100.
Examples of some of the descriptions that followed are:
Jack is a 45-year-old man. He is married and has four children. He is generally conservative, careful, and ambitious. He shows no interest in political and social issues and spends most of his free time on his many hobbies, which include home carpentry, sailing, and mathematical puzzles.
Dick is a 30-year-old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.
Subjects in these experiments tended to ignore the baseline frequency of engineers to lawyers and tried instead to determine the probability that each description described a lawyer or engineer entirely on the basis of information conveyed in the profile of each (e.g., on the personality traits listed).
In another set of experiments, referred to as the Harvard Medical School problem,Footnote 8 subjects are given the following problem:
If a test to detect a disease whose prevalence is 1/1,000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs? ____%
Subjects in these experiments tended to ignore base rates entirely and estimate that there is a 95% chance that the person has the disease.
Many experimental psychologists responded to the disconnect between certain normative models of reasoning and reasoning in natural languages by arguing that people are systematically irrational. Psychologists imported norms of reasoning from other disciplines and, when subjects did not reason in accordance with these norms, they concluded that the reasoning of the subjects is what was in error. As Massimo Piatelli-Palmarini argues, “The universal principles of logic, arithmetic, and probability calculus …. tell us what we should … think, not what we in fact think … If our intuition does in fact lead us to results incompatible with logic, we conclude that our intuition is at fault.”Footnote 9
Heuristics and Biases in Psychology
The failure of these normative models to capture reasoning in natural language led to the development of the heuristics and biases tradition in experimental psychology. Advocates of heuristics and biases claim that, rather than using normatively accepted means to solve a particular task, subjects rely upon heuristics or general rules of thumb to arrive at responses that are considered biases. A bias is defined by Jonathan Evans, Stephen Newstead and Ruth Byrne as “a systematic error, relative to the normative model provided by formal logic”Footnote 10 and as “systematic attention to a logically irrelevant feature of the task.”Footnote 11 The most notable proponents of the heuristics and biases approach are Tversky and Kahneman. However, as Thomas Sturm outlines in “The ‘Rationality Wars’ in Psychology,” many other researchers have similarly argued that this disconnect between normative models and reasoning in natural language has “bleak implications for human rationality,”Footnote 12 that “irrationality rather than rationality is the norm,”Footnote 13 and that “human reasoning is fundamentally flawed.”Footnote 14
Some of the basic tenets of the heuristics and biases tradition as described by Richard Samuels, Stephen Stich, and Michael Bishop are:
(1) People’s intuitive judgments on a large number of problems involving probability or uncertainty regularly deviate from appropriate norms of rationality.Footnote 15
(2) Many of the instances in which our probabilistic judgments deviate from appropriate norms of rationality are to be explained by the fact that, in making these judgments, people rely on heuristics like representativeness which sometimes yield reasonable judgments and sometimes lead to severe and systematic errors.Footnote 16
Results from experimental psychological tasks were imported into philosophical works—perhaps most notably in the work of Stich—to argue against the reliability of untutored reasoners and to dismiss theories that would take this natural language reasoning seriously.
In The Fragmentation of Reason, Stich argues that,
neither the neo-Goodmanian program nor any alternative program that proposes to analyze or explicate our presystemic notions of epistemic evaluation will be of any help at all in deciding whether and how their own cognitive processes or those of others might be improved.Footnote 17
In other words, Stich believes that any project that aims to take pre-theoretic intuitions about logical terms and use this data to inform normative theories about good reasoning is doomed to failure. Stich outlines the results from a number of different experiments on reasoning and claims that, “patently unacceptable rules of inference would pass the reflective equilibrium test for many people.”Footnote 18 Some of the examples Stich uses to defend this position are regression errors, affirmation bias, and the gambler’s fallacy.
In this particular case, Stich’s interdisciplinary work, rather than opening up a discipline, led to the reinforcement of pervasive biases within philosophy itself. Psychologists seeking norms of reasoning on which to base their experiments were presented with the impression that these systems of reasoning are universal; thus, when presented with deviations from these systems, psychologists concluded that the subjects were showing errors of reasoning. Philosophers then took this perceived evidence about the unreliability of untutored reasoners to support their own bias that norms of reasoning must be justified a priori or taken as foundational. Thus, the data that could call into question the universality of these systems—the experimental evidence that most reasoning does not correspond to these systems—is summarily rejected on the basis that untutored reasoners are unreliable.
In the sections that follow, I aim to disrupt this circle by presenting alternative interpretations of some of the experimental data that is taken to provide evidence of the unreliability of natural language reasoners. First, I present some experimental evidence that untutored reasoners are actually fairly reliable statistical reasoners when the statistical information is presented in the form of frequencies. This supports the general conclusion that untutored reasoners are reliable in cases where information is presented in a form with which they are already familiar. Second, I present an alternative interpretation of the results of the Wason selection task provided by Mike Oaksford and Nick Chater. Oaksford and Chater argue that subjects are not relying upon heuristics and biases in this task but, rather, are simply using a probabilistic interpretation of the conditional. I argue that subjects in these deductive tasks may be employing different systems of reasoning than those first imported from philosophers by experimental psychologists and, thus, are not failing to reason well.
The fact that there are alternative explanations for the data seen in these experimental tasks shows that philosophers and psychologists were too quick to conclude that these tasks show systematic irrationality on the part of the subjects. Philosophers were so quick to accept the conclusion that natural language users are irrational because it is convenient to believe so. It reinforces the biases within our own tradition to believe that only experts have access to rules of reasoning. I conclude by arguing that the failure of these particular normative models to correctly describe or predict reasoning in natural language could have been taken to show that the formal accounts these psychologists have imported are questionable and/or deeply contextual in nature. An alternative response to the failure of these normative models, I will argue, is to take the approach of Gerd Gigerenzer and argue that these normative models, for many different reasons, should either be replaced as models of natural language reasoning or should be recognized as deeply contextual. As Gigerenzer argues, “formal axioms and rules cannot be imposed as universal yardsticks of rationality independent of social objectives, norms and values,”Footnote 19 and “traditional axioms and rules are incomplete as behavioral norms in the sense that their normative validity depends on the social context of behavior, such as social objectives, values, and motivations.”Footnote 20 This alternative response happens to be the one I favour and have argued for elsewhereFootnote 21 but here I am simply arguing that this particular conclusion is overlooked because it broadens rather than narrows the disciplinary boundaries. While both possible conclusions about this psychological data rely upon the same philosophical background and the same psychological data, the conclusion that happens to challenge preconceptions about who has authority to determine correct reasoning is dismissed as not real philosophy. Interdisciplinarity, in this particular case, went wrong in that it locked both disciplines in a vicious circle where pervasive biases are reinforced rather than questioned. After I present arguments to motivate the alternative interpretation of this data, I will present some tentative suggestions for why interdisciplinarity failed in this particular case.
Frequencies
Experiments about frequencies can show that natural language users are more reliable statistical reasoners than was previously thought—when the statistical information is presented to them in a form that they are familiar with. Gigerenzer argues that subjects are not good at the statistical problems described earlier because they rarely encounter such tasks in everyday life. When many of these same tasks are presented as frequencies, a form with which subjects are more familiar from everyday reasoning, nearly all of the bias disappears. Leda Cosmides and John Tooby make a similar point, claiming that “our hominid ancestors were immersed in a rich flow of observable frequencies that could be used to improve decision-making, given procedures that could take advantage of them. So if we have adaptations for inductive reasoning, they should take frequency information as input.”Footnote 22 This speculation led Cosmides and Tooby to replicate the Harvard Medical School problem used by Ward Casscells, Arno Schoenberger, and Thomas Graboys, changing the problem into one in which both the input and the response required were formulated in terms of frequencies. The amended form of the Harvard Medical School problem is:
One out of every 1,000 Americans has disease X. A test has been developed to detect when a person has disease X. Every time the test is given to a person who has the disease, the test comes out positive. But sometimes the test also comes out positive when it is given to a person who is completely healthy. Specifically, out of every 1,000 people who are perfectly healthy, 50 of them test positive for the disease. Imagine that we have assembled a random sample of 1,000 Americans. They were selected by lottery. Those who conducted the lottery had no information about the health status of any of these people.
Given the information above:
on average, how many people who test positive for the disease will actually have the disease? _____ out of _____.
In sharp contrast to Casscells, Schoenberger, and Graboys, original experiment in which only 18% of subjects gave the correct Bayesian response, the above problem elicited the correct Bayesian answer from 76% of Cosmides and Tooby’s subjects. Nor is this an isolated case in which frequentist versions of probabilistic reasoning problems elicit high levels of performance. On the contrary, in many instances, when problems are framed in terms of frequencies rather than probabilities, subjects tend to reason in a manner consistent with the Bayesian normative model.Footnote 23
It has also been shown that, when the statistical information in tasks purporting to show conjunction fallacy is presented as a frequency, the conjunction fallacy disappears. For example,
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and she participated in anti-nuclear demonstrations.
There are 100 people who fit the description above. How many of them are (a) bank tellers, (b) bank tellers and active feminists?
In the answers to this format, the conjunction fallacy has been reduced from about 85% to 20% and lower.Footnote 24 We can take the failures of probabilistic reasoning in particular tasks to show that subjects are only good at statistical reasoning when the information is presented in a form with which they are familiar. The Harvard Medical School problem could be taken to clash with this conclusion since these students were already familiar with information about probabilities presented in this manner. This would show that, while natural language users are fairly reliable at statistical reasoning when the information is provided in terms of frequencies, other methods of presenting probabilities are more difficult for all natural language users to reason with—even those who have been trained in this method.
Optimal Data Selection Model
Unlike the inductive reasoning tasks, in the deductive tasks, there are no reliable ways to ensure that subjects reason in accordance with particular reasoning normsFootnote 25—if we take the norms to be classical or standard truth functional logic. However, many of the heuristics and biases that natural language users are said to employ while participating in deductive reasoning tasks can be eliminated if subjects are taken to be relying upon a different normative model for reasoning. Natural language users may be relying upon equally good norms of reasoning and not employing heuristics at all. Oaksford and Chater describe experimental data on reasoning tasks by assuming a probabilistic view of human reasoning that they call the optimal data selection model. The probabilistic model offered by Oaksford and Chater takes many of the effects and biases discovered in studies on human reasoning to be rational belief testing strategies. Most notably, they convincingly conceive of negative conclusion bias, matching bias, and confirmation bias as rational strategies for testing beliefs.
According to Oaksford and Chater, people interpret conditionals given in the Wason selection task probabilistically, which explains why subjects select the true antecedent and true consequent cards. Their argument is based upon the assumption that, in real-world reasoning, opting to turn over the true antecedent and true consequent cards is associated with the highest information gain. In other words, subjects are predisposed to analyze the conditional given in the Wason selection task probabilistically and, thus, the rational selections would be the cards that give the subject the most information.
One suggestion presented by Oaksford and Chater is that subjects mistake the Wason selection task for an open-scenario problem because they are used to reasoning in real-world situations that involve verification and not falsification. This experience primarily with real-world situations leads subjects to flip over the cards that will verify rather than falsify the conditional given. This could be why nearly all subjects flip over the potentially verifying cards A and 4 (jointly) or A (alone), but only 4% of the time flip over the potentially falsifying cards of A and 7. If we take subjects to be interpreting the conditional statement in the Wason selection task as a probabilistic conditional rather than a material conditional, then their responses can be seen as normatively correct.
While there are many issues with Oaksford and Chater’s interpretation of deductive reasoning, their work is important as it shows that subjects on deductive reasoning tasks may not be failing to reason well but may simply be reasoning in accordance with a different normative system than the researchers expect them to use. Tasks that rely upon the interpretation of conditional statements, such as the Wason selection task, may simply show that the standard truth functional conditional does not concur very well with the conditional used in everyday language. This conclusion should not be surprising to philosophers as we have long been familiar with ways in which this model of conditional truth is lacking. Philosophers examining these psychological tasks, rather than concluding that they show untutored natural language users to be poor reasoners, could instead conclude that the normative systems we take to be good models of reasoning may need to be replaced with, for example, probabilistic models.
Conclusion
Many of the experiments described above have been taken to have dire implications for human rationality. For example, Stich used the existence of the probabilistic fallacies as part of his justification for rejecting a Neo-Goodmanian project in The Fragmentation of Reason. Psychologists tend to, most optimistically, believe that these experiments show that natural language reasoning relies upon heuristics. I have given reasons above to show that such strong conclusions about the reliability of natural language reasoners and the rationality of untutored reasoners are unjustified. The experiments that rely upon statistical information presented as frequencies show that natural language users are actually fairly reliable at reasoning in forms with which they are familiar. The probabilistic interpretation of reasoning provided by Oaksford and Chater shows that subjects may not be reasoning in accordance with particular norms (basic propositional logic) but may be instead relying upon alternative, reliable forms of reasoning.
When presented with two options to interpret the psychological data, philosophers chose the side that reinforced their pervasive biases. In this case, the exchange between philosophers and experimental psychologists—while it yielded quite a lot of interesting information—resulted in a situation where the scope of what was considered relevant philosophically narrowed rather than widened. The interdisciplinarity of this work made it easier for philosophers to discount and block data that could potentially challenge pervasive biases within the field by labelling such work ‘not philosophy.’ Thus, interdisciplinary work can be useful for broadening our field and challenging our pervasive biases only if we engage with the experts and data in a particular manner. I won’t draw any firm conclusions here about how to avoid these unhealthy interdisciplinary relationships, but I will offer a few tentative suggestions about what went wrong in this particular case. One way in which interdisciplinary work can fail is if researchers explicitly search for data that reinforces a particular, already held conclusion rather than sifting through large amounts of studies with conflicting conclusions. Some of the other issues in this particular case could have been avoided by having relationships with more than one researcher in another field—specifically if those researchers hold diverging views. In most current cases of interdisciplinarity between philosophy and psychology, these methodological issues have been resolved and the connection between researchers in these areas is considerably more enriching than it was initially. Having departments and conferences specifically devoted to the interdisciplinary work in these two areas has opened up both disciplines. Thus, my general conclusion for philosophers working in interdisciplinary ways—seek out data and conclusions from the other discipline that challenge rather than reinforce the things we already think—will not be controversial or unfamiliar.
