Point 1: Repeated Participation

One of the most common concerns around MTurk samples has to do with the same individuals participating in the same or similar studies repeatedly. Although it is possible for MTurk workers to (wrongfully) participate in the same study more than once using multiple accounts, this concern has been discounted by researchers who have found little to no evidence of such activity (Chandler, Mueller, & Paolacci, Reference Chandler, Mueller and Paolacci2014). However, the possibility of MTurk workers (legitimately) participating in multiple similar studies appears to be more substantial, on which we further expand in the later section, “Nonnaïveté Among Participants.”

We view this second type of “repeated participation” issue as one of the most significant challenges for psychologists using MTurk for data collection. We completely agree with Landers and Behrend that the same individuals responding to the same set of questions over multiple times on noncognitive individual characteristics is probably not very problematic (cf., Buhrmester, Kwang, & Gosling, Reference Buhrmester, Kwang and Gosling2011; Holden, Dennie, & Hicks, Reference Holden, Dennie and Hicks2013). On the other hand, measures of cognitive ability may be more susceptible to testing or practice effects, which are generally defined as performance improvements due to multiple exposures to the same (or similar) tasks. Also, participating in a series of similar experimental studies, which are not necessarily conducted by same researchers, allows MTurk workers to potentially gain more knowledge about the underlying study objectives and research questions. Such knowledge is often said to distort the research participants’ behaviors during the experiment (i.e., “demand characteristics”). To date, no clear solutions have been offered to remedy these problems.

One additional note: Although we think that studies using MTurk samples are particularly susceptible to issues related to repeated participation, it can also be an issue in other convenience samples. For example, students in psychology and management subject pools are often asked to fill out a common set of psychological measures (e.g., Big Five personality scales) across many studies. Also, in some organizations, employees are repeatedly asked about their job satisfaction and turnover intentions multiple times in a year. Thus, testing and practice effects are not necessarily an issue unique to MTurk samples and should be considered in context.

Point 2: Compensation and Resulting Motivation

We note that the question of whether one's motivation for participating in studies would distort the study results is not inherent in MTurk (and other convenience) samples. In fact, any human subjects research is susceptible to potential response biases created by various compensation and recruiting strategies one way or another. Studies suggest that MTurk workers are willing participants who choose to partake in studies for various reasons (e.g., for fun, to kill time, to make money) and are motivated to give high-quality responses. This cannot necessarily be said for college student samples where the participation is arguably less voluntary. Although most student samples consist of individuals selecting into a specific course (e.g., introductory psychology), these students are often required to complete a certain number of studies in order to pass the course. This makes college students’ participation slightly less voluntary than that of MTurk workers who face no negative repercussions for not completing any tasks (or “HITs,” short for human intelligence tasks). Also, organizational samples vary greatly in terms of how the participation is “encouraged” or “forced” by the management, which makes it difficult to judge the underlying motivation of job incumbents for responding to survey questions and/or participating in training programs.

Although research has shown monetary incentives to be an important motivation for MTurk workers, lower payment has not been shown to impact the quality of the data (Buhrmester et al., Reference Buhrmester, Kwang and Gosling2011). While participants may be “in it for the money,” they also view MTurk as a form of entertainment and a fruitful way of spending free time (Paolacci, Chandler, & Ipeirotis, Reference Ipeirotis2010). Additionally, one study found that making a study too financially attractive (i.e., paying more than the MTurk norm) may decrease the quality of the data (Chandler et al., Reference Chandler, Mueller and Paolacci2014).

Point 3: Selection Bias

As Landers and Behrend pointed out, MTurk workers are directly restricted by the decisions to (a) sign up as MTurk workers and (b) participate in the particular study. The first decision is uniquely associated with MTurk, whereas the second decision applies to all studies using human subjects to collect data in general—regardless of the sampling design, a human subject needs to avail himself or herself to be part of the study, even when the sample was randomly drawn from the population. Therefore, we focus our discussion on the first problem, which has a fairly straightforward solution. If one's research involves psychological constructs that are reflected by the first decision of signing up as a MTurk worker (e.g., access to computers, daily computer use, comfort with online task completion, interest in getting compensated via Amazon), then choosing MTurk as a primary source of data would not be advisable. If, however, the goal is to study a diverse population of workers from various industries and regions within the United States, then MTurk workers would be preferred over college students or employees from a single organization. This point is further elaborated in the following section.

Point 4: Relevance of the Sample to Working Populations

There is a fast-growing body of research suggesting that MTurk covers a diverse range of demographic populations both in the United States and internationally (e.g., Behrend, Sharek, Meade, & Wiebe, Reference Behrend, Sharek, Meade and Wiebe2011; Buhrmester et al., Reference Buhrmester, Kwang and Gosling2011; Ipeirotis, Reference Ipeirotis2010; Paolacci et al., Reference Paolacci, Chandler and Ipeirotis2010; Ross, Zaldivar, Irani, & Tomlinson, Reference Ross, Zaldivar, Irani and Tomlinson2010; Simons & Chabris, Reference Simons and Chabris2012). Therefore, rather than a concern about MTurk, we view this fourth point (i.e., relevance of the sample to working populations) as a unique advantage that MTurk offers as a convenience sampling strategy.

Researchers who use organizational convenience samples are at risk of excluding a large portion of workers, particularly those who are marginalized, live outside the United States, and work in the informal sector. Within the United States, many employees are excluded from the scope of organizational research, including migrant workers and blue-collar workers (see Maynard & Ferdman, Reference Maynard and Ferdman2009). On the global scale, industrial and organizational psychologists are at risk of being too Western-focused (see Arnett, Reference Arnett2008) and may only study a small portion of the world's population. Fortune 500 companies, which are often studied by organizational scholars, employ close to 27 million people (Bloxham, Reference Bloxham2014)—roughly .3% of the world's population. Furthermore, studies that do examine workers outside the United States may also be limited in scope, given that many of these workers are employed in informal jobs or the informal sector. In India, for example, 68.8% of non-agricultural work takes place in the informal sector (International Labour Organization, 2011). MTurk, however, offers industrial and organizational psychologists the opportunity to study these different populations that have been largely neglected. By examining these employees, researchers are better able to understand work as it is experienced outside of a Western context and to study emerging areas in organizational research (e.g., humanitarian work psychology, cross-cultural psychology).

Nonnaïveté Among Participants

Researchers often assume naïveté among research participants; however, this may not be the case on MTurk (as well as in other samples; e.g., students participating in psychology and management subject pools do talk to one another about studies they have participated). There are two potential sources of MTurk worker nonnaïveté: communication via online discussion boards (e.g., mturkforum.com, turkernation.com) and repeated participation in similar studies posted by different researchers. The first source is likely not a critical issue, as researchers have found that the conversation on these discussion boards typically centers around payment and how interesting various studies are (Chandler et al., Reference Chandler, Mueller and Paolacci2014). The second concern, however, is highly relevant and poses a potential commons dilemma where the overuse of MTurk could result in nonnaïveté in research participants who may be self-selecting into similar types of research. Similar to our concerns above regarding the repeated participation across studies, we think that some MTurk workers may pose a potential threat to the validity of study findings by participating in psychological studies at a great frequency and also actively utilizing online forums for seeking and disseminating information about those studies.

Berinsky, Huber, and Lenz (Reference Berinsky, Huber and Lenz2012) suggest that a nontrivial proportion of the human intelligence tasks (HITs) completed on MTurk may be attributed to a group of “Super Turkers.” Pooling responses from their collaborators for a sample of 16,408 HITs, the authors found that the HITs were completed by a total of 7,498 workers. As we discussed earlier, this may or may not pose a problem, depending on the nature of the research. Nonetheless, there is evidence suggesting that the number of workers actually completing HITs is much smaller than the number of workers signed up to complete HITs on MTurk (Fort, Gilles, & Cohen, Reference Fort, Gilles and Cohen2011). While there are over 500,000 workers signed up to complete HITs on MTurk, it is unclear how many are actively completing HITs.

Does MTurk Provide High-Quality Data?