Hostname: page-component-7b7567c8c-kwt9d Total loading time: 0 Render date: 2026-11-13T10:18:23.299Z Has data issue: false hasContentIssue false
  • English
  • Français

Stop Apologizing for Your Samples, Start Embracing Them

Published online by Cambridge University Press:  28 July 2015

Xiaoyuan (Susan) Zhu,
Janet L. Barnes-Farrell and
Dev K. Dalal
Xiaoyuan (Susan) Zhu*
Affiliation:
University of Connecticut
Janet L. Barnes-Farrell
Affiliation:
University of Connecticut
Dev K. Dalal
Affiliation:
University of Connecticut
*
Correspondence concerning this article should be addressed to Xiaoyuan (Susan) Zhu, Department of Psychology, University of Connecticut, 406 Babbidge Road, Unit 1020, Storrs, CT 06269-1020. E-mail: xiaoyuan.zhu@uconn.edu
Rights & Permissions [Opens in a new window]

Extract

Landers and Behrend (2015) call for editors and reviewers to resist using heuristics when evaluating samples in research as well as for researchers to cautiously consider choosing the samples appropriate for their research questions. Whereas we fully agree with the former conclusion, we believe the latter can be extended even further to encourage researchers to embrace the strengths of their samples for understanding their research rather than simply defending their samples. We believe that samples are not inherently better or worse but rather better suited for different research objectives. In this commentary, we identify three continua on which research goals can differ to demonstrate that all samples can inform science. Depending on the position of one's research on these continua, different samples exhibit different strengths; the continua described below can be used to anchor one's sample to demonstrate how it can benefit, rather than limit, research conclusions. As discussed in the focal article, researchers will often apologize for their convenience samples as one of a litany of limitations; we hope that researchers will move sampling issues out of the limitations section and into the main discussion.

Information

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015, pp. 228 - 232
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Landers and Behrend (Reference Landers and Behrend2015) call for editors and reviewers to resist using heuristics when evaluating samples in research as well as for researchers to cautiously consider choosing the samples appropriate for their research questions. Whereas we fully agree with the former conclusion, we believe the latter can be extended even further to encourage researchers to embrace the strengths of their samples for understanding their research rather than simply defending their samples. We believe that samples are not inherently better or worse but rather better suited for different research objectives. In this commentary, we identify three continua on which research goals can differ to demonstrate that all samples can inform science. Depending on the position of one's research on these continua, different samples exhibit different strengths; the continua described below can be used to anchor one's sample to demonstrate how it can benefit, rather than limit, research conclusions. As discussed in the focal article, researchers will often apologize for their convenience samples as one of a litany of limitations; we hope that researchers will move sampling issues out of the limitations section and into the main discussion.

Generalizing to Contextualizing

Research efforts can differ with respect to whether the goal is to generalize an effect across situations or contextualize an effect to a specific setting. On one end of the continuum, researchers aim to generalize their findings across situations or organizations (i.e., external validity). The emphasis here is to understand theoretical relations and underlying mechanisms rather than test effects within a specific sample (Highhouse, Reference Highhouse2009). To the extent that contextual effects are absent—that is, the phenomenon under investigation can be expected to materialize across contexts—the specific type of sample is less relevant. When studying a fundamental psychological process applicable to multiple settings and groups of people, then a sample on this end of the continuum is advantageous, and the sample can be used to support the argument that the theoretical principles generalize across samples (Calder, Phillips, & Tybout, Reference Calder, Phillips and Tybout1981; Highhouse & Gillespie, Reference Highhouse, Gillespie, Lance and Vandenberg2009). A good example of this type of reasoning is seen in the judgment and decision-making literature wherein student and nonstudent convenience samples are used to demonstrate basic judgment and decision-making effects that manifest across samples (e.g., Schwarz & Vaughn, Reference Schwarz, Vaughn, Gilovich, Griffin and Kahneman2002).

The emphasis on the other end of this continuum is to understand whether an effect functions in a fundamentally different manner in settings with particular characteristics; that is, researchers are seeking to identify boundary conditions of a theory or effect. With the goal of further developing theory, the sample naturally becomes part of the research design as a potential moderator in order to consider how characteristics of a specific sample may alter the relations typically found. In turn, the type of sample augments the conclusions and becomes an important strength of the study. For instance, studying effects of physical proximity on interpersonal relations in settings wherein physical proximity is unconstrained, such as typical organizations, versus settings that naturally constrain proximity, such as submariners and National Aeronautics and Space Administration astronauts, makes sampling an important strength of the study. Indeed, the latter two samples allow organizational scientists to explore more nuanced effects and to provide direct and practical conclusions.

Shared Experiences to Diverse Experiences

Research ideas can span a continuum from, on the one end, requiring shared experiences to, on the other end, requiring diverse experiences to emerge. One may be interested in psychological processes that need participants to have a shared experience, such as emergence of culture or leadership. For instance, understanding how adaptive personality influences turnover in the event of organizational change (e.g., new leadership team) requires using an organizational sample wherein employees experience the same event. On the one hand, members of a sports team who experience a large-scale coaching or ownership change can be a viable proxy. On the other hand, convenience samples such as panel, crowdsourced, or snowball samples would certainly be inappropriate in this case.

Alternatively, one may be interested in how people with different experiences exhibit the same psychological processes. For example, studying people's use of heuristics during interviews should manifest across situations. Demonstrating this effect with an intentionally diverse sample can buffer against organizational/setting specific explanations and bolster the applicability of one's research findings. These types of studies would benefit from random samples of the adult workforce, to be sure, but could also be addressed with Mechanical Turk (MTurk) or online panel samples, given these participants likely vary widely in their past experiences with interviewing.

Science as a Process: Initial Stages to Later Stages

Finally, researchers should consider at which stage in the scientific process are the effects/theories they are studying, as this also speaks to the benefits of a sample. In the early stages of a research study, researchers are likely studying novel phenomena that have not been robustly observed. In other words, many early studies in an area of research are looking to establish a proof-of-concept (i.e., does this effect even exist?) to start the theory development process. Because science is advanced in incremental steps, these early studies benefit from a focused attempt at demonstrating and replicating an effect with less concern for specific samples and/or generalizing. In such studies, assuming they are appropriate for the research question, any convenience sample can be useful. In addition, these more convenient samples (e.g., undergraduate students, MTurk) facilitate randomized experiments that can control extraneous variables to establish internal validity (Hovland, Reference Hovland1959), as the initial goal is to test the theorized effect in situations in which it may not naturally occur (Highhouse, Reference Highhouse2009). The ability to test for generalizability first requires a robust understanding of the underlying principles. In this case, easy to obtain samples such as student samples, MTurk, and archival data (e.g., sports teams) appropriately facilitate investigating research questions located at the “initial stages” end of the continuum.

As the effect gets better understood, samples become more important—particularly when testing for boundary conditions of the effect. At this point, researchers can think about how their sampling decisions replicate and extend the earlier studies. In these stages, the use of random and organizational samples becomes beneficial. In addition, however, other convenience sampling techniques might be necessary. For instance, studying the adjustment of military personnel to civilian occupations likely requires some sort of snowball sampling to reach such a unique sample.

Conclusions

Although all samples have their strengths and limitations, it seems that organizational researchers are quick to apologize for their samples’ limitations rather than acknowledging what makes their samples beneficial for their studies. To be clear, samples should be considered in the design stages of a study, but default admonishment (and apologizing) for not using organizational samples is inappropriate and hinders the growth of the field. Indeed, carefully considering what samples tell us about an effect can help researchers more fully understand the phenomena we study. Although feasibility and appropriateness of using a particular sample are important to consider, they should not be confused with convenience. A sample that best answers a research question, regardless of its convenience, is the best sample (Vogt, Gardner, & Haeffele, Reference Vogt, Gardner and Haeffele2012). Researchers should embrace the utility of their samples, especially when one can bolster the case for their sample by matching it to the purpose of their research on the continua provided above and other continua that are not discussed here. Again, we are not suggesting that one should ignore the limitations of a sample; rather, we suggest critically evaluating the potential advantages of a sample and utilizing them to strengthen the research conclusions.

References

Calder, B. J., Phillips, L. W., & Tybout, A. M. (1981). Designing research for application. Journal of Consumer Research, 8, 197–207.Google Scholar
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554–566.CrossRefGoogle Scholar
Highhouse, S., & Gillespie, J. Z. (2009). Do samples really matter that much? In Lance, C. E. & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 249–268). New York, NY: Routledge.Google Scholar
Hovland, C. I. (1959). Reconciling conflicting results derived from experimental and survey studies of attitude change. American Psychologist, 14, 8–17.CrossRefGoogle Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Schwarz, N., & Vaughn, L. A. (2002). The availability heuristic revisited: Ease of recall and content of recall as distinct sources of information. In Gilovich, T., Griffin, D., & Kahneman, D. (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 103–119). Cambridge, United Kingdom: Cambridge University Press.Google Scholar
Vogt, P. W., Gardner, D. C., & Haeffele, L. M. (2012). Sampling, selection, and recruitment. When to use what research design (pp. 126–129). New York, NY: Guilford Press.Google Scholar