The field of experimental philosophy has received considerable attention, essentially for producing results that seem highly counter-intuitive and at the same time question some of the fundamental methods used in philosophy. Much of the debate that this field has generated focuses on the role of intuitions in philosophy. This debate pre-dates the emergence of experimental philosophy; however, the findings of experimental philosophers have given the debate a new urgency. One of the major contributions of experimental philosophy in the discussion on the role of intuitions in philosophy has been concrete evidence in support of intuitional diversity, i.e. the idea that intuitions vary systematically depending on variables such as ethnicity, socioeconomic status, gender, or age. (Weinberg et al. Reference Weinberg, Nichols and Stich2001; Colaço et al. Reference Colaço, Buckwalter, Stich and Machery2014; Buckwalter and Stich forthcoming).
The paper by Weinberg and colleagues titled ‘Normativity and Epistemic Intuitions’ published in 2001 in large part gave rise to the movement now known as experimental philosophy. The authors of the paper presented data showing that the epistemic intuitions of East and South Asian individuals differed significantly from that of their ‘Western’ counterparts on a host of scenarios. Since the publication of this paper, ‘[o]ne of the most widely discussed kinds of intuitional diversity has been cultural diversity,’ namely the hypothesis that ‘our philosophical intuitions seem to be sensitive to our own cultural background’ (Alexander Reference Alexander2012: 72). In addition to data on cultural diversity, Weinberg et al. (Reference Weinberg, Nichols and Stich2001) also provided data indicating that individuals from different socioeconomic backgrounds have differing epistemic intuitions.
Weinberg et al. (Reference Weinberg, Nichols and Stich2001) has been very influential and the topic of extensive discussion. However, despite its influence and reach, the findings presented in the paper have not been tested for their reproducibility but typically simply accepted as fact. We collected data through four different sources, two on-line and two in-person where we presented individuals with scenarios identical in wording to those asked by Weinberg et al. (Reference Weinberg, Nichols and Stich2001). Our results strongly suggest that epistemic intuitions are not significantly different among individuals from different ethnic or socioeconomic backgrounds. Given our findings, we have to conclude that the results of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) are not reproducible and hence the notion of intuitional diversity for these cases among different ethnic and socioeconomic groups needs to be corrected.
We will advance as follows: The next section gives a brief introduction on the role of intuitions in philosophy. We keep this section brief because it is only meant to put the significance of the findings discussed in this paper in perspective. Given the central role that intuitions play in philosophy, this issue has been the subject of extensive debate and any attempt at providing a comprehensive review here would not do the topic justice. For some of the discussions that took place before the emergence of experimental philosophy, see Wild (Reference Wild1938), Stich (Reference Stich1988), Ramsey (Reference Ramsey1992), Bealer (Reference Bealer1996, Reference Bealer2000), Cummins (Reference Cummins, DePaul and Ramsey1998), Goldman and Pust (Reference Goldman, Pust, DePaul and Ramsey1998), Gopnik and Schwitzgebel (Reference Gopnik, Schwitzgebel, DePaul and Ramsey1998), Gutting (Reference Gutting, DePaul and Ramsey1998), Kornblith (Reference Kornblith, DePaul and Ramsey1998), Shafir (Reference Shafir, DePaul and Ramsey1998), E. Sosa (Reference Sosa, DePaul and Ramsey1998), Wisniewski (Reference Wisniewski, DePaul and Ramsey1998), Osbeck (Reference Osbeck1999) and for some of these debates in the context of experimental philosophy, see Stich (Reference Stich2001), Weinberg et al. (Reference Weinberg, Nichols and Stich2001), Nichols (Reference Nichols2004), Levin (Reference Levin2005), Williamson (Reference Williamson2004), D. Sosa (Reference Sosa2006), Goldman (Reference Goldman2007), Alexander and Weinberg (Reference Alexander and Weinberg2007), Gendler (Reference Gendler2007), E. Sosa (Reference Sosa2007, Reference Sosa, Bishop and Murphy2009), Dowell (Reference Dowell2008), Feltz (Reference Feltz2008, Reference Feltz2009), Liao (Reference Liao2008), Symons (Reference Symons2008), Mallon et al. (Reference Mallon, Machery, Nichols and Stich2009), Alexander et al. (Reference Alexander, Mallon and Weinberg2010), Cullen (Reference Cullen2010), Deutsch (Reference Deutsch2009, Reference Deutsch2010), Wright (Reference Wright2010), Chudnoff (Reference Chudnoff2011), Alexander (Reference Alexander2012), Cappelen (Reference Cappelen2012), Nagel (Reference Nagel2012), Ichikawa (Reference Ichikawa, Booth and Rowbottom2014), Buckwalter and Stich (forthcoming). Section 2 examines ethnic differences. Subsection 2.1 provides a description of our four data sources and the methods and materials used in data collection. In subsection 2.2 we present our results for East Asian and Western participants and compare these to the outcomes of Weinberg et al. (Reference Weinberg, Nichols and Stich2001). In subsection 2.3 we present the results for South Asian and Western participants and again compare these to the relevant data from Weinberg et al. (Reference Weinberg, Nichols and Stich2001). Section 3 examines intuitional differences based on socioeconomic status. Section 4 concludes with a discussion.
1. INTUITIONS AND PHILOSOPHY
Intuitions play a unique role in philosophy that is very different from other disciplines. Levin (Reference Levin2005: 194), for example, writes that the use of intuitions ‘has been characteristic, perhaps definitive, of philosophical argumentation throughout its history’. Goldman (Reference Goldman2007: 1), explains that ‘[o]ne thing that distinguishes philosophical methodology from the methodology of the sciences is its extensive and avowed reliance on intuition’. In the philosophical literature, intuitions are (broadly understood) simply ‘spontaneous judgments’ to philosophical questions (Stich Reference Stich2001). For example, when presented with a scenario, a reader's immediate reaction as to whether something was morally permissible or not counts as an intuition. Often, individuals do not immediately have an explanation for these reactions. For example, a person may judge one action as permissible and another as impermissible, without being able to point out the relevant features (e.g. intention) that led to differences in judgments. A common description of the use of intuitions in philosophy is as follows: philosophers construct thought experiments and test their intuitive responses. These intuitions serve as data points that are used to substantiate or challenge theories. Within this framework ‘the role and corresponding epistemic status of intuitional evidence in philosophy is similar to the role and corresponding epistemic status of perceptual evidence in science’ (Alexander Reference Alexander2012: 11). There are numerous descriptions of this method and although these vary somewhat, an account of intuition as data or evidence lies at the core. We will refer to this approach to philosophy as the Intuition as Evidence approach (IAE) following Liao (Reference Liao2008).
To fill in IAE further, consider the following two descriptions: Stich (Reference Stich2001), who traces this methodology to Plato, writes that philosophers proceed
to test normative claims against people's spontaneous judgments about real and hypothetical cases. Contemporary philosophers often call these spontaneous judgments ‘intuitions.’ If the normative claim and people's intuitions agree, the claim is vindicated. But if […] a normative principle conflicts with people's intuitions, then something has to give. Sometimes we may hold on to the normative claim and ignore a recalcitrant intuition. But if a normative principle conflicts with lots of intuitions or […] if it conflicts with an intuition that we would be very reluctant to give up, then Plato's method requires that we reject the principle and try to come up with another one. (Stich Reference Stich2001)
Goldman (Reference Goldman2007) writes that
[t]o decide what is knowledge, reference, identity, or causation […], philosophers routinely consider actual and hypothetical examples and ask whether these examples provide instances of the target category or concept. People's mental responses to these examples are often called ‘intuitions’ and these intuitions are treated as evidence for the correct answer. At a minimum, they are evidence for the examples' being instances or non-instances of knowledge, references, causation, etc. Thus, intuitions play a particularly critical role in a certain sector of philosophical activity. (A. I. Goldman, Reference Goldman2007: 1)
An exemplary instance of IAE often cited is Gettier's Reference Gettier1963 paper titled ‘Is Justified True Belief Knowledge?’. At the time of Gettier's writing – and further back to Plato (Burnyeat Reference Burnyeat1990) – typical accounts equated knowledge with justified true belief. Gettier provided two thought experiments wherein although an individual was described to have justified true belief, readers did not have the intuition that this individual actually had knowledge. Bealer (Reference Bealer1996) summarizes that ‘at one time many people accepted the doctrine that knowledge is justified true belief. But today we have good evidence to the contrary, namely, our intuitions that situations like those described in the Gettier literature are possible and that the relevant people in those situations would not know the things at issue’ (Bealer Reference Bealer1996: 122).
What made Gettier's examples convincing was that the intuitions they elicited were shared widely. Goldman (Reference Goldman2007: 2) notes that it ‘was the fact that almost everybody who read Gettier's examples shared the intuition that these were not instances of knowing. Had their intuitions been different, there would have been no discovery’. A degree of uniformity of intuitions is of special importance for IAE because without it the practice would be on extremely unstable grounds. The equivalent of this in scientific practice would be if data from experiments were to vary depending on who carried out the experiments. Systematically differing intuitions based on individuals' backgrounds such as ethnicity, socioeconomic status or gender are often referred to as intuitional diversity. As mentioned before, one of the major contributions of experimental philosophy to general philosophy has been the (alleged) concrete evidence in support of intuitional diversity. These findings, if robust, would make it very problematic for philosophers to rely on intuitions as a source of evidence. If intuitions were to vary in this way, they would merely be reflective of cultural or socioeconomic background or gender rather than be informative on details of philosophical scenarios; this is the suggestion that Weinberg et al. (Reference Weinberg, Nichols and Stich2001) advance.
Given the important implications of the findings of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) for philosophical methodology, we set out to examine the experiments more closely. In our paper we focus almost exclusively on Weinberg et al. (Reference Weinberg, Nichols and Stich2001) and carry out an exact replication of its experiments. There are several reasons for focusing on Weinberg et al. (Reference Weinberg, Nichols and Stich2001). First, at least according to some, Weinberg et al. (Reference Weinberg, Nichols and Stich2001) gave rise to the movement now known as experimental philosophy (Koppl Reference Koppl2011; Williamson Reference Williamson2011) and it may simply be of historical interest if the findings that gave rise to this (now very popular) movement were non-reproducible.
Second, and more importantly, Weinberg et al. (Reference Weinberg, Nichols and Stich2001) has been very influential, currently with a citation count of 417.Footnote 1 In addition to references in philosophy journals, the paper has also been cited in economics, law, and mathematics journals (Pardo Reference Pardo2005; Löwe et al. Reference Löwe, Müller, Müller-Hill, Van Kerkhove, De Vuyst and Van Bendegem2009; Koppl Reference Koppl2011). The paper also appeared in the edited volume ‘Experimental Philosophy’ which is itself a popular and influential outlet. To give a sense of the status this paper enjoys, Knobe (Reference Knobe2007: 82) in discussing intuitional diversity writes that ‘[p]erhaps the most celebrated work in this vein is Weinberg, Nichols and Stich's paper on intuitions about knowledge’. As another example, Doris (Reference Doris2005: 674) writes that ‘‘Experimental philosophy’ pertaining to various topics is now – happily – appearing with increasing frequency […] The locus classicus is Weinberg, Nichols, and Stich (Reference Weinberg, Nichols and Stich2001)'.
Because of this wide exposure there are numerous instances in published papers and books citing Weinberg et al. (Reference Weinberg, Nichols and Stich2001) as evidence that East and South Asian individuals have different epistemic intuitions from their ‘Western’ counterparts. This difference in intuitions is often treated as an established fact. For example, Bishop and Trout (Reference Bishop and Trout2005: 705) write that ‘[i]n a fascinating study, Weinberg, Nichols, and Stich (Reference Weinberg, Nichols and Stich2001) found that people in different cultural and socioeconomic groups make significantly different epistemic judgments’. Buckwalter (Reference Buckwalter2010: 396) notes that ‘[f]amously, Weinberg et al. Reference Weinberg, Nichols and Stich2001 shows that […] divergence of intuition extends to the epistemic domain’. Mallon et al. (Reference Mallon, Machery, Nichols and Stich2009: 340) write that ‘Weinberg et al. found that there are indeed systematic cross-cultural differences in epistemic intuitions’. There are numerous other examples of papers citing Weinberg et al. (Reference Weinberg, Nichols and Stich2001) as evidence for intuitional diversity; for some of these, see Alexander and Weinberg (Reference Alexander and Weinberg2007), Knobe (Reference Knobe2007), Nadelhoffer and Nahmias (Reference Nadelhoffer and Nahmias2007), Knobe & Nichols (Reference Knobe and Nichols2008), Feltz (Reference Feltz2009), Zamzow and Nichols (Reference Zamzow and Nichols2009), Beebe and Buckwalter (Reference Beebe and Buckwalter2010), Alexander (Reference Alexander2012). Furthermore, in conversations with colleagues, we have also often heard this supposed difference in epistemic intuitions between different ethnic groups being treated as fact.
Aside from this widespread acceptance of the results, many philosophers have engaged with the findings of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) and much effort has been expended in attempting to explain why the findings do not bear heavily on IAE or why despite the results, IAE is sufficiently robust as an approach to withstand intuitional diversity. For some of these discussions, see Weatherson (Reference Weatherson2003), Williamson (Reference Williamson2004, Reference Williamson2011), Liao (Reference Liao2008), Deutsch (Reference Deutsch2009, Reference Deutsch2010), Grundmann (Reference Grundmann2010), Shieber (Reference Shieber2010), Nagel (Reference Nagel2012, Reference Nagel2013). However, responses have typically taken the findings of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) as a given and started their replies from there.
The final reason for carrying out an exact replication of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) is the following. Experimental philosophy uses the tools of experimental psychology to study questions of interest to philosophers. By adopting these tools, philosophers have inevitably adopted some of the shortcomings of psychology as well. Various fields of psychology currently face a ‘crisis of confidence’ which amounts to an overproduction of false-positive results in the published literature (Ioannidis Reference Ioannidis2005, Reference Ioannidis2012; Bakker et al. Reference Bakker, van Dijk and Wicherts2012; Pashler and Harris Reference Pashler and Harris2012; Pashler and Wagenmakers Reference Pashler and Wagenmakers2012). There are several reasons for this, one of them being a lack of interest by researchers to conduct replications and relatedly an aversion by journals to publishing this kind of work (Bozarth and Roberts Reference Bozarth and Roberts1972; Neuliep and Crandall Reference Neuliep, Crandall and Neuliep1991; Makel et al. Reference Makel, Plucker and Hegarty2012; Nosek et al. Reference Nosek, Spies and Motyl2012). We hope that by providing these data from our exact replication we can contribute in small part to preventing experimental philosophy from going down a similar path to some of the areas of psychology. We hope to show that there is an important role for replications in the scientific process. Theoretical descriptions of the scientific process typically place a high value on carrying out replications and consider reproducibility integral to science (Braude Reference Braude1979; Lamal Reference Lamal and Neuliep1991; Collins Reference Collins1992; Popper Reference Popper2002; Francis Reference Francis2012; Nosek et al. Reference Nosek, Spies and Motyl2012). However, this importance granted to replications in theory, does not generally translate into practice (Smith Reference Smith1970; Amir and Sharon Reference Amir, Sharon and Neuliep1991; Hendrick Reference Hendrick and Neuliep1991; Collins Reference Collins1992; Makel et al. Reference Makel, Plucker and Hegarty2012). Many psychologists agree that replications are critical in lowering the rate of false positives (Amir and Sharon, Reference Amir, Sharon and Neuliep1991; Koole and Lakens Reference Koole and Lakens2012; Nosek et al. Reference Nosek, Spies and Motyl2012; Pashler and Harris Reference Pashler and Harris2012; Ritchie et al. Reference Ritchie, Wiseman and French2012) and calls for more replications have been made frequently during various crises of the past decades, however, in practice these calls have mostly remained unanswered – with some notable exceptions (Nosek Reference Nosek2012) – and in experimental philosophy systematic replications were completely lacking until we started some of our efforts.
Aside from what we believe to be the significance of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) and why we believed it warranted a full replication of its experiments, the authors of that paper themselves underline the importance of their work by noting that their evidence, if robust,Footnote 2 shows that ‘a sizeable group of epistemological projects – a group which includes much of what has been done in epistemology in the analytic tradition – would be seriously undermined’ (Weinberg et al. Reference Weinberg, Nichols and Stich2001: 429). Furthermore, one of the co-authors writes that ‘[i]ndeed, in light of these new findings some philosophers – I am one of them – have come to think that after 2400 years it may be time for philosophy to stop relying on Plato's method’ (Stich Reference Stich2001). These are strong claims (by the authors' own admission) that need to be examined carefully. This paper is an attempt at a careful examination that we believe should have been carried out many years ago, before all the discussion that Weinberg et al. (Reference Weinberg, Nichols and Stich2001) prompted, before all the back and forth between proponents and opponents of the views expressed in the paper and before all the effort was spent discussing the claims of the paper.
2. ETHNICITY AND EPISTEMIC INTUITIONS
2.1 Methods and materials
The experiments for this paper were conducted between February 2011 and March 2012. A first draft of this paper was uploaded to SSRN on October 29, 2012 and first drafts were sent to conferences starting in July 2012.
2.1.1 Scenarios
Below are the scenarios as we presented them to participants. All scenarios were taken from Weinberg et al. (Reference Weinberg, Nichols and Stich2001).
Car Case
Bob has a friend, Jill, who has driven a Buick for many years. Bob therefore thinks that Jill drives an American car. He is not aware, however, that her Buick has recently been stolen, and he is also not aware that Jill has replaced it with a Pontiac, which is a different kind of American car. Does Bob really know that Jill drives an American car, or does he only believe it?
REALLY KNOWS ONLY BELIEVES
We used the same wording in all of our surveys except for in Data set 4 where we replaced the names of the cars from Buick and Pontiac to Toyota and Honda, respectively and also changed the origin of the cars from ‘American’ to ‘Japanese’, accordingly.
Individualistic Truetemp Case
One day Charles is suddenly knocked out by a falling rock, and his brain becomes re–wired so that he is always absolutely right whenever he estimates the temperature where he is. Charles is completely unaware that his brain has been altered in this way. A few weeks later, this brain re-wiring leads him to believe that it is 71 degrees in his room. Apart from his estimation, he has no other reasons to think that it is 71 degrees. In fact, it is at that time 71 degrees in his room.
Does Charles really know that it was 71 degrees in the room, or does he only believe it?
REALLY KNOWS ONLY BELIEVES
In addition to the Individualistic Truetemp case, Weinberg et al. (Reference Weinberg, Nichols and Stich2001) also collected data on two variations named as ‘Elders’ and ‘Community Wide Truetemp’ scenarios. We did not collect data on these scenarios as Weinberg and colleagues themselves report no significant differences between different ethnic groups on these cases.
Conspiracy Case
It's clear that smoking cigarettes increases the likelihood of getting cancer. However, there is now a great deal of evidence that just using nicotine by itself without smoking (for instance, by taking a nicotine pill) does not increase the likelihood of getting cancer. Jim knows about this evidence and as a result, he believes that using nicotine does not increase the likelihood of getting cancer. It is possible that the tobacco companies dishonestly made up and publicized this evidence that using nicotine does not increase the likelihood of cancer, and that the evidence is really false and misleading. Now, the tobacco companies did not actually make up this evidence, but Jim is not aware of this fact. Does Jim really know that using nicotine doesn't increase the likelihood of getting cancer, or does he only believe it?
REALLY KNOWS ONLY BELIEVES
Zebra Case
Pat is at the zoo with his son, and when they come to the zebra cage, Pat points to the animal and says, ‘that's a zebra.’ Pat is right – it is a zebra. However, given the distance the spectators are from the cage, Pat would not be able to tell the difference between a real zebra and a mule that is cleverly disguised to look like a zebra. And if the animal had really been a cleverly disguised mule, Pat still would have thought that it was a zebra. Does Pat really know that the animal is a zebra, or does he only believe that it is?
REALLY KNOWS ONLY BELIEVES
2.1.2 Data sets
Throughout the rest of this paper we will use the terms and abbreviations Western (W), East Asian (EA) and South Asian or Indian Subcontinent (SC), following the terminology in Weinberg et al. (Reference Weinberg, Nichols and Stich2001) for consistency.
Data set 1
Procedure
For this data set we visited undergraduate classes at the London School of Economics (LSE). Participation was voluntary although no one refused. After a brief introduction, we handed out a one-page questionnaire. Each student only saw one scenario. We explained that there were several different questionnaires and that therefore some would complete the questionnaire faster than others. We did hand out several different questionnaires but only one of them included a scenario surveying epistemic intuitions (the Car case). In all, the whole process took about five minutes.
We mainly visited philosophy classes, but, given the relatively small size of the philosophy department, we also visited some classes in the International Relations department to complement the data. About 14 percent of the data came from non-philosophy classes. We will provide a breakdown of the numbers in the results section. There was no significant difference between data collected in philosophy and non-philosophy classes with p-exact = 0.557 for N = 153.
Participants
We will provide the number of participants that fell into each one of the categories that we analyzed (EA, SC and W) since this is the focus of our discussion. Data set 1 consisted of 41 EAs, 35 SCs and 79Ws for a total sample of 155. For the exact criteria used to categorize participants for all of the data sets, see Appendix A.
Scenarios presented
In this data set we only presented the Car case to participants.
Data set 2
Procedure
For our second study we used the resources at the London School of Economics' Behavioural Research Lab (BRL). The BRL compiles a database of individuals interested in participating in studies. Participants then receive email notifications whenever studies are being conducted. Individuals received 5 pounds sterling to participate in a 30-minute study that consisted of several different tasks including answering questions from a wide variety of different fields in philosophy. Upon arrival participants were given a brief introduction. Then they were brought to a workstation in a computer lab where they started the survey.
Participants
This data set consisted of 60 Ws, 60 EAs and 59 SCs for a total sample of 179.
Scenarios presented
We surveyed the Truetemp and Conspiracy cases for this data set.
Data set 3
Procedure
For the third data set we launched questionnaires on SurveyMonkey (SM) that consisted of six questions, the four described in Section 2.1.1 and two on semantic intuitions taken from Machery et al. (Reference Machery, Mallon, Nichols and Stich2004). Participants sign up with SM and receive links to surveys from time to time. For every survey completed, SM donates $0.50 to a charity of the participant's choice. In addition, participants are entered into a draw for a chance to win a $100 gift card of an online retailer.Footnote 3 The first page of the survey was a brief introduction giving some background information. This included, for example, that the survey was for an academic study and the approximate time the study would take. After seeing the six questions, participants filled out a demographic questionnaire and finally there was also a text box to leave comments.
Participants
This data set consisted of 75 Ws, 36 EAs and 12 SCs for a total of 123.
Scenarios presented
We tested all four scenarios in this data set. We did not carry out significance tests for the SC samples, as these were too small.
Data set 4
Procedure
The data for this study was collected through Harvard University's Moral Sense Test (MST) website.Footnote 4 Participants visited the MST website without being solicited and took part in the surveys that consisted of several different tasks and the Car scenario was included as a filler question. Some of the tasks included watching video clips or visualizing certain situations.
Participants
This sample consisted of 193 Ws and 15 SCs. Given the small sample of SCs, we mainly include this data set for completeness, as there was a statistical difference on the Car scenario.
Scenarios presented
We only tested the Car case here. We did not have sufficient EAs here to carry out a meaningful comparison.
2.2 Results of East Asians and Westerners
When comparing EAs and Ws, Weinberg and colleagues found statistically significant differences for the Car and Truetemp scenarios, however, they failed to find differences for the other two cases. In our replication attempts, we did not attain a significant difference for any of the scenarios. Table 1 presents a summary of the results, including the results of Weinberg et al. (Reference Weinberg, Nichols and Stich2001). In all tables, * denotes p < 0.05; ** denotes p < 0.01.
a Data Set 1 was collected in philosophy as well as political science classes. This row presents data collected in philosophy classes only.
Note 1: Weinberg et al. mention in their section on South Asians that there were no differences between EAs and Ws for the Conspiracy and Zebra cases, however, they do not provide any further details of sample sizes or p-values.
The Car scenario produced the clearest disparity between the original and replication studies. Whereas Weinberg and colleagues report that a majority of East Asian individuals had the ‘Really Knows’ intuition, none of the replication studies reproduced this finding. In fact, in one of the replication studies (Data set 3) the percentage of ‘Really Knows’ answers was slightly lower for East Asian than for Western participants. Although the percentage of ‘Really Knows’ answers for EAs was higher than for Ws in Data set 1, the difference was nowhere as extreme as the result that Weinberg et al. (Reference Weinberg, Nichols and Stich2001) reports.
With regard to the other scenarios, we did not detect a difference for Truetemp, whereas Weinberg et al. did, and Weinberg et al. themselves did not detect any differences for the last two scenarios – Conspiracy and Zebra – and neither did we; that is, there was no disparity between the original and replication studies.
2.3 South Asians and Westerners
With the exception of the Truetemp case, Weinberg et al. report significant differences between SCs and Ws for all of the four scenarios. Our findings, again, paint a different picture (Table 2).
Note 2: Weinberg and colleagues do not specify whether there was or was not a significant difference for this scenario.
Note 3: We did not have sufficient data for this scenario in any of our data sets to carry out a meaningful comparison.
Summary of the results for socioeconomic status.
Note 4: Weinberg and colleagues do not state explicitly whether the Car and Truetemp scenarios yielded a significant difference or whether no data was collected. The implication seems to be that data were collected but no difference was detected.
The Car case, again, produced the biggest difference between the original and replication studies. The outcome of the Car case Weinberg et al. present for SCs and Ws is similar to the sample of EAs and Ws. In both cases a larger number of non-Western participants respond that Bob really knows that Jill drives an American car, whereas this relationship is reversed for Western participants. Western individuals, as opposed to non-Westerners, according to the original paper predominantly choose the ‘Only believes’ answer choice. Data set 1, where data were collected in classrooms and was closest to the original paper in procedures yielded a very different outcome. The percentage of South Asian and Western participants were almost identical. We did attain a significant difference between these two groups for Data set 4, however, as mentioned before the SC sample size was small and the outcome may not be very meaningful; we mainly included this data for completeness.
3. SOCIOECONOMIC STATUS AND EPISTEMIC INTUITIONS
In their section on socioeconomic backgrounds Weinberg et al. (Reference Weinberg, Nichols and Stich2001: 453) concludes that socioeconomic status (SES) has a ‘major impact on subjects’ epistemic intuitions'. As a reason for why individuals from different SES may have different epistemic intuitions, Weinberg and colleagues write that a ‘possibility is that high SES subjects accept much weaker knowledge-defeaters than low SES subjects because low SES subjects have lower minimum standards for knowledge’ (Weinberg et al. Reference Weinberg, Nichols and Stich2001: 447). The authors continue that ‘whatever the explanation turns out to be, the data we've reported look to be yet another serious embarrassment for the advocates of [IAE]’ (Weinberg et al. Reference Weinberg, Nichols and Stich2001: 447). Our replication attempts do not support this conclusion.
For this part of the paper, we set up two questionnaires on SurveyMonkey (SM) to test the epistemic intuitions of individuals from different socioeconomic backgrounds on the same scenarios for which Weinberg et al. (Reference Weinberg, Nichols and Stich2001) report differences.
3.1 Methods
Scenarios
We used the same wording as in Weinberg et al. (Reference Weinberg, Nichols and Stich2001) for all the scenarios with the exception of the Car case where we replaced the names of the cars from Buick and Pontiac to Ford and Jeep, respectively, in order to make the scenario more current.Footnote 5 For the wording of the scenarios, see Section 2.1.1.
Procedure
The SM procedure was the same as described in Section 2.1.2 of this paper. Participants sign up with SM and receive links to surveys from time to time. For every survey completed SM donates $0.50 to a charity of the participant's choice. In addition, participants are entered into a draw for a chance to win a $100 gift card of an online retailer.
We set up two templates on SM, which we will refer to as Template 1 and Template 2 from here on. The templates were identical with the exception of the order in which the scenarios were presented. Participants first saw a brief introduction stating that we were conducting the questionnaire for an academic research project in the field of philosophy. Next, participants saw the four scenarios from Section 2.1.1. For the first template the order was Conspiracy, Zebra, Truetemp, and Car. In the second template the order was Zebra, Car, Conspiracy, and Truetemp. The survey concluded with a very short demographic section where we asked about ethnic background and education.Footnote 6 SM furthermore provided us data on gender, age range, household income,Footnote 7 and education. For our data analysis we used data on education that participants submitted in our surveys and not data provided by SM. There was some discrepancy between the two sources, which may be partly explained by the fact that the information is not always up to date with SM and individuals make progress in their educational attainments.
Weinberg et al. (Reference Weinberg, Nichols and Stich2001) reported significant differences for the Conspiracy and the Zebra cases (from their paper, it appears that the other two scenarios did not yield a difference, although this is not mentioned explicitly). Hence, we chose the specific sequence mentioned above in order to have the Conspiracy case as the first scenario in Template 1 and the Zebra case as the first scenario in Template 2.
Participants
Weinberg and colleagues used an education proxy to categorize participants as either low or high socioeconomic status.Footnote 8 Individuals who indicated that they had never attended college were classified as low SES, whereas participants who indicated that they had taken one or more courses at the college level were classified as high SES. We used the same criteria in classifying participants.
The survey with the second template was initiated about two weeks after the first survey and we asked SM not to send out invitations to any of the individuals who participated in the first questionnaire. For the first template, we asked SM to restrict participation to individuals who were 24 years of age or older. We were concerned that given the criteria for distinguishing low and high SES by an education proxy we might get many young respondents for the low SES group. After reviewing the data for the first template, we realized that our concern was unfounded and we omitted this requirement for the second template.
For Template 1 our sample consisted of 107 participants (38 low SES, 69 high SES). For Template 2 our sample consisted of 134 individuals (47 low SES, 87 high SES).
3.2 Results
Template 1
For Template 1 none of the scenarios yielded a significant difference with the exception of the Car case.
Template 2
In creating Template 2, we made some changes to the first template. First, we changed the order in which the scenarios were presented. Since Weinberg et al., in addition to the Conspiracy case also reported a significant difference for the Zebra scenario, we wanted this case to be placed at the beginning, so we could rule out order effects. Second, given that there was a significant difference for the Car case in our first template we wanted to place this scenario further toward the beginning of the survey in order to rule out participation fatigue as the reason for the difference.
There are several things worth pointing out here. First, the Zebra case again did not yield a significant difference when presented as the first scenario. In fact, this time none of the scenarios yielded a significant difference. The Car case produced the closest p-value to a significant level (i.e. 0.177); however, this time the direction of the responses was reversed when compared with the first template. This time low SES participants had a lower percentage of ‘Really Knows’ responses than high SES participants, which contradicts Weinberg et al.'s explanation about the role of socioeconomic status on epistemic judgments.
Further analyses
There were various other tests we ran to examine the data. First, we ran an analysis of the combined data from the two templates. Despite the large sample (N = 240), none of the scenarios produced a significant outcome or a p-value close to 0.10. We do not think that this is merely because of cancelling order effects (the Car case was significant in Template 1, however, the direction was reversed in Template 2 and these effects could be cancelling each other out when combining the data). Rather it seems to be that despite the increased sample size, we still could not find a difference between the two groups. We tested for order effects by comparing the data of the two templates and the only scenario that produced a significant difference was the Car case.
Next, we wanted to see if there would be a significant difference between the two groups if we made the difference in educational attainment greater. So, for the high SES group we included in our next analysis only participants who had at least completed their Bachelor's degree. Low SES was coded as before. The outcomes (statistical significance) for the four scenarios did not change for either one of the templates.
We further ran an analysis excluding participants where the self-reported education level and that provided by SM did not match. None of the outcomes changed. Finally, we ran analyses excluding all participants who fell in the age range 18–29. This made the Car case for the second template significant (again, in the opposite direction of Template 1) but otherwise all other outcomes remained unchanged.
4. DISCUSSION AND CONCLUDING REMARKS
Discussion
When discussing our results at conferences or informally with colleagues, we have been told on multiple occasions that given that differences between East Asian and Western individuals have been shown for some cognitive tasks in the Nisbett literature (Nisbett et al. Reference Nisbett, Peng, Choi and Norenzayan2001), it should not come as a surprise that these ethnic groups may exhibit different epistemic intuitions. There are two responses that we can offer. First, differences in cognitive tasks in some areas, do not necessarily predict differences in other areas. Secondly, and much more importantly, although the work of Nisbett and colleagues has been influential, there have been some notable failures of replication in that field (Rayner et al. Reference Rayner, Li, Williams, Cave and Well2007, Reference Rayner, Castelhano and Yang2009; Zhou et al. Reference Zhou, Gotch, Zhou and Liu2008; Evans et al. Reference Evans, Rotello, Li and Rayner2009; Lu et al. Reference Lu, Daneman, Reingold, Love, McRae and Sloutsky2008; Miellet et al. Reference Miellet, Zhou, He, Rodger and Caldara2010); as it stands more work is needed to attain a better picture.
Furthermore, there have been two studies that can be considered failures of conceptual replications of Weinberg et al. (Reference Weinberg, Nichols and Stich2001), namely Nagel et al. (Reference Nagel, Juan and Mar2013) and Turri (Reference Turri2013). Neither one of these papers tested the exact same scenarios as Weinberg and colleagues; nevertheless, these studies failed to reproduce the same effect. Finally, at the time of preparing this manuscript's final changes, we were contacted by a group who informed us that they tested the Car scenario (exact replication) in the northeast United States and this group, too, was unable to reproduce the effect reported by Weinberg and colleagues (Minsun and Yuan, ms). In all, there is now increasing evidence that the findings of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) are not reproducible. The main reason we see for why there is a difference between the original and replication studies and hence the failure of replications is that the sample sizes in the original study are relatively small; on average the sample size of the EA/SC and low SES samples is 24.
Weinberg and colleagues draw some strong conclusions from their study. For example, one of the co-authors writes that ‘[h]igh SES Americans and low SES Americans have different epistemic intuitions! Moreover, in many cases these differences are quite dramatic.’ Similarly, Stich writes that a ‘reasonable conclusion is that philosophy's 2400 year long infatuation with Plato's method has been a terrible mistake’ (Stich Reference Stich2001). The simple takeaway of our study is that much more and better evidence is needed to make such strong claims.
For the most part of this paper we attempted to remain neutral on the debate concerning the role of intuitions in philosophy. We wanted the focus to be on the findings presented; however, our findings and the results of the three groups just mentioned above, all weaken one of the main arguments experimental philosophy has brought against the use of intuitions in philosophy, in as far as these arguments rely on Weinberg et al. (Reference Weinberg, Nichols and Stich2001).
It may be the case that had we surveyed individuals born and residing in East and South Asia, we may have detected responses different from Westerners.Footnote 9 However, there are several things worth pointing out in this regard. Turri (Reference Turri2013) surveyed Indians living in India on a Gettier-type scenario and although the form of presentation and the scenario were different from Weinberg et al. (Reference Weinberg, Nichols and Stich2001), Turri found much lower rates of ‘Really Knows’ answer choices (15%) and also no difference when compared with an American population, 96% of whom indicated English as their native language. For the exact details, see Turri (Reference Turri2013). Minsun and Yuan (ms) compared EAs and Ws as categorized on the basis of native language on the Car scenario and found no significant difference (Minsun and Yuan, ms). These studies indicate that there may not be a difference between Ws and East/South Asians born and residing in their native countries. However, strictly speaking this possible difference cannot be ruled out, as we do not have the necessary data.
This discussion, though, is beyond the scope of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) and consequently the current paper. Weinberg et al. surveyed a population living and studying in the New Jersey area; the target population was ethnic minorities living in the West and for these they found big differences when compared with individuals of European descent. This is mainly what makes Weinberg et al.'s results so surprising. It would not be too surprising if surveying a population that is very unlike Ws yielded different outcomes, as one is likely to introduce problems of language and task comprehension, amongst others. What gives Weinberg et al. (Reference Weinberg, Nichols and Stich2001) its surprise factor is that merely surveying participants from different ethnic backgrounds within the same university yielded different intuitive responses. And demographic variables such as ethnicity, most philosophers believe, should not factor into our evaluation of philosophical questions.
Conclusion
The aim of this paper was to test the robustness of the results of Weinberg et al. (Reference Weinberg, Nichols and Stich2001). Despite collecting data from various sources and attaining larger samples in several of the cases, we failed to detect differences on epistemic intuitions between participants from different ethnic backgrounds and socioeconomic statuses. With regard to socioeconomic status, we collected data from 241 individuals on four scenarios surveying epistemic intuitions for which Weinberg et al. (Reference Weinberg, Nichols and Stich2001) report significant differences (on two of the cases) but failed to find statistically significant differences. Given these data, we do not believe that socioeconomic status by itself has an impact on epistemic intuitions for the cases evaluated in this paper. With regard to ethnicity and epistemic intuitions, even though we collected data in several different settings, we could not replicate the results of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) on differences among individuals from East Asian, South Asian and Western backgrounds. Given these data, we do not believe that ethnic background has a significant impact on epistemic intuitions.
As mentioned in the Introduction, Weinberg et al. (Reference Weinberg, Nichols and Stich2001) has been an influential paper, which has received numerous citations. In discussions with other researchers in the field it often appears that it is an established fact that epistemic intuitions differ among ethnic groups. Our data suggest that this conception needs to be corrected. Despite the important implications of the original paper and despite the debate surrounding the findings of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) for conducting epistemology as well as philosophy in general, there had not been exact replication attempts of Weinberg et al. (Reference Weinberg, Nichols and Stich2001) to test the robustness of the reported results until 2011 when we started out on this project. As mentioned above, since then more work has been done in this area and all studies produced results in line with our findings. One of our initial hopes for sharing our data was that other researchers would study the cases, so that the original results could be verified independently. We further hope that other researchers will find it worthwhile to examine these cases and we hope to have provided a first and useful reference point with this paper.Footnote 10
APPENDIX
Data set 1
In order to determine participants' ethnic backgrounds, we used the relevant questions from Richard Nisbett's demographic instrument. We used this questionnaire as Weinberg and colleagues describe using this method; however, in the interest of keeping the survey short, we did not use all of the questions; for example, we did not ask about SAT or ACT scores or annual salary, but instead restricted our questions mostly to those specifically aimed at identifying ethnic background. These included the following questions: self-identified ethnicity (if the response to this was white/Caucasian, we further inquired about specific origin, i.e. Eastern Europe, Middle East/West Asia), native language, place of birth, place of birth of parents and grandparents.
In order to be classified as W participants had to self-identify as white/Caucasian and be born in the EU (there were no participants from Iceland or Switzerland) or US (there were no participants from Australia or Canada). We furthermore asked about family's background and excluded participants who were of Eastern European and generally non-Western (for example, West Asian) background. At least one parent and two grandparents had to have EU or US as their place of birth. In all, only eight participants did not indicate all their parents and grandparents to have been born in the EU/US. Excluding these participants from analysis did not change the outcomes and in fact increased p-values. Two participants mistakenly indicated their birthplace as 1991 (their birth year); we included these among Ws since all their grandparents and parents were born in the EU or US and their native language was English.
In order to be classified as EA or SC participants had to self-identify as East Asian (China, Korea, Japan) or South Asian (Bangladesh, India, Pakistan – there were no participants from Nepal), respectively, in the ethnicity part of the questionnaire. Furthermore, at most one parent and one grandparent could have been born in the EU or US. Among EAs none of the participants had any parents or grandparents who were born in the EU or US; among SCs four individuals had one parent born in EU/US and one participant had exactly one grandparent born in EU/US. As before, excluding these participants did not change the final outcomes and in fact increased p-values. For the EA sample, nine participants were born in the West (Australia, the UK or US); however, excluding these participants increased p-values slightly. For the SC sample, out of 35 participants (there was one non-response), 23 were born in the West (all in the UK), leaving only 12 participants born internationally. Comparing the latter sample with the W sample is not very meaningful because of the small sample size. Nevertheless, a comparison did not yield a significant difference, although p-values did decrease.
Data set 2
We tested two scenarios in this data set, namely Truetemp and Conspiracy. For Truetemp, we used the same criteria as in Data set 1 to categorize participants. There were two Canadian individuals in this sample who were born in Canada with native language English/French whom we included as W. None of the EA and SC individuals had either a parent or grandparent born in the EU or US with the exception of one EA who indicated one grandparent to have been born in the EU/US. This participant was born in China with Chinese as her/his native language. For the EA sample, two participants were born in the West (the UK, US); excluding these from analysis did not change significance. Five individuals in the SC sample were born in the West (all EU); excluding these from analysis did not make a difference and p-exact remained at 1.000.
For Conspiracy there was a problem with the database and about half of the participants' birth places were not recorded. We used the other pieces of information to categorize participants. As before participants had to self-identify as East Asian or South Asian to be included in these categories. None of the EA and SC individuals had any parents or grandparents born in the EU or US with the exception of one individual who indicated that one of their parents and one of their grandparents were born in the EU or US. For the available data, no EA participants were born in the West and in the SC sample only two were born in the West. Excluding these from analysis did not change significance.
Data set 3
SM collects demographic information on individuals who sign up to participate. We asked SM to send out invitations to people of white/Caucasian background and individuals of Asian background. SM does not classify among different regions of Asia, so we used our own demographic questionnaire to filter for East and South Asian participants. Being in SM's white/Caucasian category did not automatically categorize respondents as Westerners. For example, West Asians who were in SM's white/Caucasian category were not classified as Western. We relied on our own questionnaire to categorize participants, however, we used SM's categorization to narrow down the target audience. Ethnicity was self-identified as a response to the question ‘How would you describe your ethnic background?’ Additionally, we asked for native language and used this information for categorizing participants as well. For example, if someone self-identified as East Asian, however, indicated their native language as Hindi or Vietnamese, this person was classified as South Asian or South-east Asian, respectively. Only one participant among the white/Caucasian group indicated their native language as non-English (it was German), all others indicated English as their native language. Participants who self-identified as EA and indicated English as their native language were classified as EA.
Data set 4
For this data set we used similar criteria to Data set 3; we used self-identification to categorize participants and further used information on birthplace and native language wherever available to correct for obvious mistakes.
