Background

The question of whether student samples can be used to study psychological phenomena has been a source of debate throughout the history of behavioral science research (e.g., Farber, Reference Farber1952; Mook, Reference Mook1983; Oakes, Reference Oakes1972; Sears, Reference Sears1986). And, it has been especially contentious in applied areas such as consumer, political, and organizational psychology (Calder, Phillips, & Tybout, Reference Calder, Phillips and Tybout1981; Kardes, Reference Kardes1996; Locke, Reference Locke and Locke1986).

Farber (Reference Farber1952) provided an early forceful defense of student samples, noting that students are only inappropriate for studying phenomena at a descriptive level. When one is doing inferential research, according to Farber, the college student is as indispensable to psychological research as the fruit fly is to the geneticist. Given that most I-O research is aimed at identifying general principles that can be applied across organizational settings, we are interested in understanding and controlling causal mechanisms. If undergraduate students are among the people covered by a theory, then they are indeed an appropriate sample on which to test the theory (see Highhouse & Gillespie, Reference Highhouse, Gillespie, Lance and Vandenberg2009, for an elaboration of this argument). College students, however, continue to be viewed among I-O psychologists as a different species from working people (Aguinis & Bradley, Reference Aguinis and Bradley2014; Shen et al., Reference Shen, Kiger, Davies, Rasch, Simon and Ones2011). This is despite the fact that, comparing across subfields of psychology, laboratory results from I-O psychology [typically using students as subjects] most reliably predicted field results (Mitchell, Reference Mitchell2012).

Like Goes With Like

The historic resistance to student samples for conducting applied psychological research is a classic example of overusing the representativeness heuristic (Highhouse & Gillespie, Reference Highhouse, Gillespie, Lance and Vandenberg2009; Kardes, Reference Kardes1996). The representative heuristic is a reflexive tendency to assess the similarity of things along superficial dimensions and to expect things to conform to the rule that “like goes with like” (Gilovich, Griffin, & Kahneman, Reference Gilovich, Griffin and Kahneman2002). A doctor, for example is more likely to make a diagnosis of a disorder if the patient seems similar to the prototypical patient with that disorder. Similarly, we expect research on organizations to come from studies using members of an organization rather than from studies using college students.

Given the persistence of the representativeness heuristic, student samples are unlikely to overcome their stigmatization in the review process. There is, however, hope for MTurk to replace college students as the new fruit fly for I-O research. This is because research using MTurk workers is more similar to research using members of an organization than it is to research using college students. On the basis of the commonly cited advantages of real-world samples (e.g., Sears, Reference Sears1986), Table 1 compares traditional college students, MTurk workers, and typical organizational members used in research perceived to be generalizable (see Buhrmester, Kwang, & Gosling, Reference Buhrmester, Kwang and Gosling2011; Paolacci & Chandler, Reference Paolacci and Chandler2014).

Table 1

Commonly Cited Advantages of Real-World Samples

As you can see, MTurk workers have none of the perceived limitations of confused, mercurial, and partially formed college students. As such, there should be no reason for reviewers and editors to dismiss MTurk workers on the basis of external validity—especially if they are not dismissing members of a single organization on that same basis.

Resuscitating Experimentation in I-O

Today's I-O research base is predominately correlational (i.e., passive observation) in nature (Greenberg & Tomlinson, Reference Greenberg and Tomlinson2004; Highhouse, Reference Highhouse2009). Experimentation within I-O psychology—which heretofore has been unable to shake its historical association with college student samples—is becoming a lost art. We will not reiterate the advantages of experimentation for inferring causality (see Colquitt, Reference Colquitt2008; Stone-Romero, Reference Stone-Romero and Rogelberg2002). Our goal is to merely suggest that MTurk provides a unique opportunity in I-O to bring back random assignment to conditions. The use of the Internet provides considerable opportunities to immerse people in realistic situations and gather information on thoughts, feelings, attitudes, and intentions (Aguinis & Bradley, Reference Aguinis and Bradley2014). With MTurk, there is no interaction with the experimenter, which avoids experimenter demand effects. And, there is lower risk of a contaminated sample, as one might get from students or coworkers talking among themselves (Paolacci, Chandler, & Ipeirotis, Reference Paolacci, Chandler and Ipeirotis2010).

Experiments are designed with the goal of generalizing theoretical explanations beyond the specific experimental circumstances, and passive observation can be used to identify boundary conditions of the theoretical explanation as it is directly applied to specific organizational circumstances (Brinberg & McGrath, Reference Brinberg and McGrath1985; Fromkin & Streufert, Reference Fromkin, Streufert and Dunnette1976). Good applied research takes advantage of both random experimentation and passive observation.

Concluding Thoughts

We agree with Landers and Behrend (Reference Landers and Behrend2015) that MTurk workers are highly appropriate for studying workplace phenomena. Moreover, we believe that the use of such samples represents a unique opportunity to reinvigorate experimentation within I-O psychology. As Landers and Behrend noted, quality of data is not contingent on its difficulty to acquire. By embracing MTurk, we are opening new avenues for discovery in our field.