Comparing MTurk With Other Data-Collection Techniques for Sampling From Hard-To-Reach Populations

With the advent of the Internet, the ways in which researchers recruit participants and administer surveys and experiments have changed dramatically. One particularly new method for data collection is Amazon's MTurk, in which individuals can opt-in to research studies in return for nominal compensation. One benefit of utilizing MTurk is that it maintains participants’ anonymity and confidentiality. As such, MTurk has a variety of strengths that lend themselves particularly well to recruiting participants who belong to hard-to-reach groups, especially in comparison with other forms of data collection traditionally used to sample from these populations.

As outlined by Landers and Behrend (Reference Landers and Behrend2015), a great deal of organizational research relies on samples collected from student populations or from single organizations. Student samples may be useful because they are easy to reach and require relatively little compensation. Despite these benefits, student samples are not likely to comprise many employees, particularly individuals from special populations (e.g., LGBT, immigrant populations). Also, student samples tend to be more homogenous than are random national samples in that the participants tend to be mostly White and between the ages of 18 and 22 (e.g., Eisenberg & Wechsler, Reference Eisenberg and Wechsler2003). Organizational samples ensure that data are more representative of employee experiences than student-based responses are, but these samples are also limited in the extent to which they comprise special populations.

Other researchers have reached special populations of employees through recruitment via community-based organizations (Campbell et al., Reference Campbell, Adams and Patterson2008), via snowball sampling techniques (Law, Martinez, Ruggs, Hebl, & Akers, Reference Law, Martinez, Ruggs, Hebl and Akers2011), or by randomly approaching participants in a densely traveled public area (McCormack, Reference McCormack2014). Using MTurk may provide solutions to potential threats to validity and generalizability associated with each of these techniques. With respect to using community-based organizations and snowball sampling, the data are traditionally obtained through personal contacts and connections, which presents limitations as outlined in the focal article. Furthermore, the researchers often do not control who is invited to participate in the research when these strategies are utilized. Thus, the veracity of the identities of the participants cannot be confirmed, and the participants may all share a common characteristic or be located in a common geographical region. Furthermore, these strategies require participants to publicly self-identify as the identity of interest, which may result in sampling bias (e.g., only those who identify strongly will be contacted). With respect to randomly approaching participants, these participants will necessarily be geographically constrained, and they may or may not be employees.

A particular strength of MTurk in collecting data from hard-to-reach populations is that it affords complete anonymity to participants. As a consequence, it is easier to collect data on participants who represent highly stigmatized, potentially concealable identities (e.g., HIV+, psychological disabilities) as these participants would be more likely to reveal their true identities when asked in such an anonymous format. Indeed, previous scholars have noted the benefits to collecting data through online and computer-based survey methodologies due to the fact that participants are more comfortable in sharing private information through these anonymous platforms (Gosling, Vazire, Srivastava, & John, Reference Gosling, Vazire, Srivastava and John2004; Levine, Ancill, & Roberts, Reference Levine, Ancill and Roberts1989; Locke & Gilbert, Reference Locke and Gilbert1995). Along these lines, researchers have also found more diversity in responses related to sexual behaviors and health-related issues when using these anonymous computer-based reporting techniques (Turner et al., Reference Turner, Ku, Rogers, Lindberg, Pleck and Sonenstein1998).

Past Research That Has Utilized MTurk To Sample From Hard-To-Reach Populations

Although very few studies have used MTurk to sample from hard-to-reach populations, the studies that have been conducted suggest that this is a viable method of accessing these individuals. A study by Tenenbaum, Byrne, and Dahling (Reference Tenenbaum, Byrne and Dahling2014) recruited a sample of 153 individuals with physical disabilities in the United States by paying each participant $0.80. The authors recorded the type of disability, the age of onset, and the severity of disability of participants and then examined the independent and interactive effects of these factors on the RIASEC self-efficacy inventory (Lenox & Subich, Reference Lenox and Subich1994). The authors utilized several screening procedures to ensure the participants had a physical disability. The final sample included a large number of unique disabilities, such as mobility limitations, amputations, inflammatory diseases, and sensory injuries.

Vaughn, Cronan, and Beavers (Reference Vaughn, Cronan and Beavers2015) collected data on MTurk from individuals who identified as LGB by using quota sampling to ensure equal representation among groups and offering participants $0.50 in exchange for participation. In particular, these authors advocate for more research to establish the veracity of using MTurk to sample from LGB populations. Papa, Lancaster, and Kahler (Reference Papa, Lancaster and Kahler2014) examined three different types of loss: bereavement, job loss, and divorce. Data were collected from 424 MTurk participants who met the following inclusion criteria: loss of a job for at least 6 months, death of a close loved one, or divorce within the last 12 months. Upon completion of the survey, participants were compensated $1.50. The authors report that the demographic results of this study were similar to other MTurk studies.

In Carr's (Reference Carr2014) study of adult cancer survivors (n = 111), she utilized a two-part study design with the explicit intention to examine the viability of utilizing MTurk as an effective sampling technique. For the initial survey, participants were paid $0.50, and for the followup, they were paid $0.70. This study found that participants were honest in their responses (as determined using a malingering scale), test–retest reliabilities were high, and approximately 90% of participants fully completed both surveys. This study represents one of the more rigorous examinations of the viability of MTurk to sample from hard-to-reach populations, and this should be replicated in other samples.

Finally, one of the authors recently collected qualitative data from cancer survivors using both MTurk and in-person surveys during a Susan G. Komen cancer fundraiser race. This study examined disclosure of cancer history in interview contexts. These data demonstrate that the MTurk participants reported disclosing at relatively similar rates as their non-MTurk counterparts. Due to the open-ended nature of the data, statistical comparisons cannot be reported; however, substantive differences between the results of these two sampling techniques were inconsequential.

Advantages of Using MTurk To Sample From Hard-To-Reach Populations

In general, MTurk has demonstrated several potential strengths in recruiting participants for organizational research, as outlined by the focal article. Again, participants recruited in this way are typically compensated much less than traditional panels are, and they represent employees from a variety of different locations and types of organizations or industries. Studies that utilize data from a single workplace limit the ability for researchers to examine the impact of contextual factors (e.g., diversity climate, perceived organizational support) on the experiences of minority employees; therefore, researchers have collected data from multiple organizations to employ cross-level analyses across organizational units (Gonzalez & DeNisi, Reference Gonzalez and DeNisi2009; McKay, Avery, & Morris, Reference McKay, Avery and Morris2009) to examine such issues. However, collecting samples from multiple organizations, if even possible, can be time-consuming and expensive. Also, a call has been made to incorporate data from smaller organizations, as most organizational research focuses exclusively on very large and profitable organizations (Storey, Reference Storey1994). Indeed, studying small businesses is important as this sector represents nearly 55 million jobs (48.5% of all jobs; U.S. Census Bureau, 2011). Additionally, with regard to diversity-related outcomes, many companies within this sector are so small that they are not required to abide by Equal Employment Opportunity Commission regulatory policies. Thus, it is important to understand the experiences of employees within these less-regulated organizations. We argue that MTurk may excel as a data-collection method in such circumstances by providing data from a large number and variety of organizations.

Additionally, users can restrict survey settings in MTurk to filter participants by certain characteristics. For example, MTurk can filter participants by IP address in order to restrict access to only U.S. participants. We recommend that researchers utilize this feature when conducting studies where variables of interest are about U.S. minorities (e.g., stereotypes that are culture-specific). Additionally, results of a recent study by Feitosa, Joseph, and Newman (Reference Feitosa, Joseph and Newman2015) found measurement invariance on a Big Five personality measure across responses in an organizational field sample and an MTurk sample, but only when the sample was restricted to U.S. IP addresses. Thus, researchers need to consider both ecological validity (i.e., are participants representative of the population of interest?) and whether cultural perspectives impact differences in how respondents interpret the measures themselves.

Potential Problems (and Solutions) Associated With Using MTurk To Sample From Hard-To-Reach Populations

Although it is clear that there are several benefits to using MTurk samples for collecting data on diverse, hard-to-reach participants, there are also some potential concerns, specifically with regards to the ability to lie about one's demographic background (Skitka & Sargis, Reference Skitka, Sargis and Amichai-Hamburger2005). Indeed, it is true that the increased anonymity afforded to MTurk users and the fact that participants are paid for taking part in the studies increases the chance that they may lie about their identities in order to participate in the study and receive payment. While we do acknowledge that this is a concerning issue regarding the use of MTurk data pools for studying hard-to-reach populations, we propose several potential remedies for this problematic issue, namely, (a) screening participants before stating the eligibility and purpose of the study, (b) filtering participants out of the study or into different studies based on their individual characteristics or identities, and (c) offering modest payments for study participation.

First, a novel way to ensure respondents are not lying about being a member of a minority group (that the authors are currently using with success) is to ask participants to self-identify before they know the purpose of the research. The participants are first asked to respond to a short series of demographic questions, with a question identifying them as part of the group of interest embedded into the beginning of the survey. These questions should be presented before the participants learn about the explicit purpose of the study, so that they are not primed with any particular manner in which to respond. For example, if interested in gathering a sample of workers who identify as LGB, one question may ask the respondent to report their sexual orientation from a list of possible options. Those who are not a member of the group of interest (e.g., self-identify as heterosexual) could be excluded from full participation. This technique would help to thwart participants from successfully fabricating an identity in order to participate and receive payment because they would not know what identity is required to qualify for participation in the study. This idea is promising, yet some researchers may feel hesitant to reject such a large number of participants in this way.

Thus, a second possible strategy is to collect data for two studies simultaneously: one that requires a specific minority group and one that does not. Similar to the technique described above, participants would be presented with a question regarding the specific demographic characteristic of interest (e.g., a question asking participants to indicate their sexual orientation). Then, those participants who do identify with the minority group will be directed to the primary study, whereas those who do not identify with that minority group will be directed to a different study. Of course, this method could only be used when the base rate for the minority sample of interest is not too low. Otherwise, this method would quickly become too costly. One solution to this problem could involve careful monitoring of the participation rates and coordinating multiple secondary studies when enough participants have been filtered through. Also, this technique would require that data-collection on the second study not be adversely affected by not having participants from the specific minority group in question. This strategy, however, may be especially beneficial for collecting dual perspectives from both stigmatized targets and nonstigmatized employees (e.g., collecting data using LGB participants for the primary study and filtering non-LGB participants into a study specific to heterosexual individuals).

Third, researchers should provide monetary incentives that are appealing but not overly enticing to participants, to reduce the likelihood that participants will lie about their characteristics in order to participate. There is typically an abundance of studies for participants to complete, and thus, providing modest to low compensation for participation and telling participants that they must belong to a specific identity group will likely reduce the chances that participants would lie in order receive payment. This strategy is supported by previous research showing that the amount of compensation provided for completion was not related to response quality. That is, paying more does not necessarily mean that responses will be of higher quality, but it may be related to longer data-collection periods (Buhrmester, Kwang, & Gosling, Reference Buhrmester, Kwang and Gosling2011), especially when searching for highly specific groups.

In general, other best practices should be taken into account when using data collected from MTurk, such as incorporating manipulation checks, establishing thresholds for missing data, ensuring that there is enough variability in responses, deleting responses with identical IP addresses, and matching key demographic characteristics (Gosling et al., Reference Gosling, Vazire, Srivastava and John2004). These strategies would allay fears brought forth in the focal article on repeated participation and missing data.