Henrich et al. show that the vast majority of research in the behavioral sciences continues to be based on populations the authors call WEIRD because they are unlikely to be representative of humankind. Even more alarmingly, much of the research published in top-tier journals is not even representative of the populations in WEIRD (Western, Educated, Industrialized, Rich, and Democratic) countries. For example, in the 510 samples published in the Journal of Personality and Social Psychology (JPSP) in 2002, 85% of them were student samples, 71% of the participants were female, more than 80% were white, and the mean age was 22.9 years (Gosling et al. Reference Gosling, Vazire, Srivastava and John2004).
What should we do about this? Henrich et al. conclude their article by urging institutions to improve the infrastructure for collecting data from non-WEIRD samples as well as the incentives for studying them. However, Henrich et al. offer very little in the way of concrete practical suggestions for expanding the reach of research in the behavioral sciences.
We propose that the Internet holds great promise for broadening the participant base of research in the behavioral sciences (Gosling & Johnson Reference Gosling and Johnson2010; Reis & Gosling Reference Reis, Gosling, Fiske, Gilbert and Lindzey2010). Using the Internet, researchers can deliver to participants a broad range of graphics, photographs, and dynamic media (Krantz & Williams Reference Krantz, Williams, Gosling and Johnson2010); obtain informant reports (Vazire Reference Vazire, Gosling and Johnson2010); and administer surveys (Tuten Reference Tuten, Gosling and Johnson2010), questionnaires (Johnson Reference Johnson, Gosling and Johnson2010), ability tests (Schroeders et al. Reference Schroeders, Wilhelm, Schipolowski, Gosling and Johnson2010), and experiments (Reips & Krantz Reference Reips, Krantz, Gosling and Johnson2010). Participants can be randomly assigned to experimental conditions, reaction times can be measured, and a broad range of incentives for participation can be offered (Göritz Reference Göritz, Gosling and Johnson2010).
Internet methods offer researchers many advantages over traditional methods in terms of improved efficiency, accuracy, cost effectiveness, and reach (Gosling & Johnson Reference Gosling and Johnson2010; Gosling et al. Reference Gosling, Vazire, Srivastava and John2004; Reis & Gosling Reference Reis, Gosling, Fiske, Gilbert and Lindzey2010). But how do Internet samples fare regarding Henrich et al.'s concerns about generalizability and representativeness? We do know that Internet samples are generally more diverse than the “traditional” samples in the top psychology journals with respect to gender, socioeconomic status, geographic region, and age (Gosling et al. Reference Gosling, Vazire, Srivastava and John2004). But do such samples offer any hope of reaching the kinds of participants that Henrich et al. rightly note are so under-represented in the behavioral sciences?
To find out, we examined a dataset of personality questionnaires collected in Dutch, English, German, or Spanish via the Internet in 2009 (for details of the method, see Srivastava et al. Reference Srivastava, John, Gosling and Potter2003). We examined only those 564,502 participants (81%) who had indicated the country in which they resided and were aged between 9 and 90. We followed the four contrasts used by Henrich et al.: (1) people from modern industrialized societies versus small-scale societies; (2) people from Western societies versus non-Western industrialized societies; (3) people from the United States versus other Western societies; and (4) university-educated versus nonuniversity-educated people from the United States.
With respect to the first comparison, our sample almost certainly fails to capture individuals living in small-scale societies, which is unsurprising given the written format and the languages used. So to provide some insight into the extent to which Internet samples capture participants beyond modern industrialized societies, we examined the percentage of our participants who were from the 34 countries classified by the International Monetary Fund (IMF) as “advanced economies” (WEO Groups and Aggregates Information 2009): 19% (N = 104,928) were not from these advanced economies.
With respect to the second contrast, we found that 20% (N = 111,962) of the sample were from non-Western societies (i.e., not from North America, Israel, Australia, New Zealand, and Western Europe). With respect to the third contrast, within these Western societies, 35% (N = 158,300) were not from the United States.
With respect to the fourth contrast, we did not have information on whether the participants were college students. However, we do know that 66% (N = 369,916) of the sample (who ranged in age between 9 and 90) were not from the 18 to 22 age group that characterizes the vast majority of college students. With respect to diversity in terms of ethnicity, a substantial 41% were non-white.
All four contrasts suggest that Internet samples are not as dominated by WEIRD participants as are samples currently published in behavioral science journals. Moreover, even though the percentages of non-WEIRD participants in the Internet samples may seem modest, Internet methods permit the collection of large samples, so the absolute sample sizes of non-Weird participants can be quite impressive. For example, although the sample was predominantly North American (52% from the United States and 5% from Canada), the sample represented a breadth of geographic regions from around the world: 111 countries, from Albania (N = 215) to Venezuela (N = 1,920), were represented in the sample by at least 100 participants each.
Our data were collected using a website that did not specifically target non-WEIRD samples and used Western languages (Dutch, English, German, Spanish) predominant in WEIRD countries; thus, the findings almost certainly underestimate the percentages of non-WEIRD participants that could be obtained in studies targeting non-WEIRD participants and using languages common in non-WEIRD samples.
Like all methods, Internet-based methods are subject to limitations. The samples are certainly not representative of humankind, participants must have access to the Internet and know how to use a Web browser, and some studies cannot be delivered online (Johnson & Gosling Reference Johnson, Gosling, Gosling and Johnson2010). However, the global spread of Web access will diminish the sampling biases. Moreover, current infrastructure and technology already provide the means for distributing compensation to participants (e.g., via Amazon.com's Mechanical Turk) and for accessing populations that do not have their own Web access (e.g., via laptop computers, tablets, and smart-phones administered by local research assistants, even those with minimal technical knowledge).
No single method can rectify the field's over-reliance on WEIRD samples. However, Internet methods are one promising tool that should be used in combination with others to combat the problem of WEIRD research. Our analyses suggest that Internet samples can help ameliorate the biases found in typical research samples and can help wean WEIRD researchers off WEIRD samples.
Henrich et al. show that the vast majority of research in the behavioral sciences continues to be based on populations the authors call WEIRD because they are unlikely to be representative of humankind. Even more alarmingly, much of the research published in top-tier journals is not even representative of the populations in WEIRD (Western, Educated, Industrialized, Rich, and Democratic) countries. For example, in the 510 samples published in the Journal of Personality and Social Psychology (JPSP) in 2002, 85% of them were student samples, 71% of the participants were female, more than 80% were white, and the mean age was 22.9 years (Gosling et al. Reference Gosling, Vazire, Srivastava and John2004).
What should we do about this? Henrich et al. conclude their article by urging institutions to improve the infrastructure for collecting data from non-WEIRD samples as well as the incentives for studying them. However, Henrich et al. offer very little in the way of concrete practical suggestions for expanding the reach of research in the behavioral sciences.
We propose that the Internet holds great promise for broadening the participant base of research in the behavioral sciences (Gosling & Johnson Reference Gosling and Johnson2010; Reis & Gosling Reference Reis, Gosling, Fiske, Gilbert and Lindzey2010). Using the Internet, researchers can deliver to participants a broad range of graphics, photographs, and dynamic media (Krantz & Williams Reference Krantz, Williams, Gosling and Johnson2010); obtain informant reports (Vazire Reference Vazire, Gosling and Johnson2010); and administer surveys (Tuten Reference Tuten, Gosling and Johnson2010), questionnaires (Johnson Reference Johnson, Gosling and Johnson2010), ability tests (Schroeders et al. Reference Schroeders, Wilhelm, Schipolowski, Gosling and Johnson2010), and experiments (Reips & Krantz Reference Reips, Krantz, Gosling and Johnson2010). Participants can be randomly assigned to experimental conditions, reaction times can be measured, and a broad range of incentives for participation can be offered (Göritz Reference Göritz, Gosling and Johnson2010).
Internet methods offer researchers many advantages over traditional methods in terms of improved efficiency, accuracy, cost effectiveness, and reach (Gosling & Johnson Reference Gosling and Johnson2010; Gosling et al. Reference Gosling, Vazire, Srivastava and John2004; Reis & Gosling Reference Reis, Gosling, Fiske, Gilbert and Lindzey2010). But how do Internet samples fare regarding Henrich et al.'s concerns about generalizability and representativeness? We do know that Internet samples are generally more diverse than the “traditional” samples in the top psychology journals with respect to gender, socioeconomic status, geographic region, and age (Gosling et al. Reference Gosling, Vazire, Srivastava and John2004). But do such samples offer any hope of reaching the kinds of participants that Henrich et al. rightly note are so under-represented in the behavioral sciences?
To find out, we examined a dataset of personality questionnaires collected in Dutch, English, German, or Spanish via the Internet in 2009 (for details of the method, see Srivastava et al. Reference Srivastava, John, Gosling and Potter2003). We examined only those 564,502 participants (81%) who had indicated the country in which they resided and were aged between 9 and 90. We followed the four contrasts used by Henrich et al.: (1) people from modern industrialized societies versus small-scale societies; (2) people from Western societies versus non-Western industrialized societies; (3) people from the United States versus other Western societies; and (4) university-educated versus nonuniversity-educated people from the United States.
With respect to the first comparison, our sample almost certainly fails to capture individuals living in small-scale societies, which is unsurprising given the written format and the languages used. So to provide some insight into the extent to which Internet samples capture participants beyond modern industrialized societies, we examined the percentage of our participants who were from the 34 countries classified by the International Monetary Fund (IMF) as “advanced economies” (WEO Groups and Aggregates Information 2009): 19% (N = 104,928) were not from these advanced economies.
With respect to the second contrast, we found that 20% (N = 111,962) of the sample were from non-Western societies (i.e., not from North America, Israel, Australia, New Zealand, and Western Europe). With respect to the third contrast, within these Western societies, 35% (N = 158,300) were not from the United States.
With respect to the fourth contrast, we did not have information on whether the participants were college students. However, we do know that 66% (N = 369,916) of the sample (who ranged in age between 9 and 90) were not from the 18 to 22 age group that characterizes the vast majority of college students. With respect to diversity in terms of ethnicity, a substantial 41% were non-white.
All four contrasts suggest that Internet samples are not as dominated by WEIRD participants as are samples currently published in behavioral science journals. Moreover, even though the percentages of non-WEIRD participants in the Internet samples may seem modest, Internet methods permit the collection of large samples, so the absolute sample sizes of non-Weird participants can be quite impressive. For example, although the sample was predominantly North American (52% from the United States and 5% from Canada), the sample represented a breadth of geographic regions from around the world: 111 countries, from Albania (N = 215) to Venezuela (N = 1,920), were represented in the sample by at least 100 participants each.
Our data were collected using a website that did not specifically target non-WEIRD samples and used Western languages (Dutch, English, German, Spanish) predominant in WEIRD countries; thus, the findings almost certainly underestimate the percentages of non-WEIRD participants that could be obtained in studies targeting non-WEIRD participants and using languages common in non-WEIRD samples.
Like all methods, Internet-based methods are subject to limitations. The samples are certainly not representative of humankind, participants must have access to the Internet and know how to use a Web browser, and some studies cannot be delivered online (Johnson & Gosling Reference Johnson, Gosling, Gosling and Johnson2010). However, the global spread of Web access will diminish the sampling biases. Moreover, current infrastructure and technology already provide the means for distributing compensation to participants (e.g., via Amazon.com's Mechanical Turk) and for accessing populations that do not have their own Web access (e.g., via laptop computers, tablets, and smart-phones administered by local research assistants, even those with minimal technical knowledge).
No single method can rectify the field's over-reliance on WEIRD samples. However, Internet methods are one promising tool that should be used in combination with others to combat the problem of WEIRD research. Our analyses suggest that Internet samples can help ameliorate the biases found in typical research samples and can help wean WEIRD researchers off WEIRD samples.