Why Is Little Attention Paid to Sampling?

There are many reasons why we have not focused much on sampling in I-O psychology. First, we would like to add to the list of reasons offered by Landers and Behrend by stipulating that little emphasis on sampling may be due to a lack of clear criteria regarding what constitutes “good” or even “acceptable” sampling. For example, Rynes described “better” datasets as having “larger sample sizes, higher response rates, better measures, and fewer errors” (Rynes, Reference Rynes and Kozlowski2012, p. 410) and repetitively referred to larger samples throughout her chapter as if that were the criterion for “better.” However, does larger necessarily mean better? We need large enough samples to avoid Type II errors. But beyond that, we suggest that large samples are not useful if the larger numbers of people, organizations, or other sample elements are not representative of the population about which we aim to make conclusions.

Second, we go further than Landers and Behrend to suggest that I-O psychologists don't emphasize sampling in the peer review process enough. Our evidence for this claim is based on a literature review we recently conducted in which we examined articles published between 2011 and 2013 in Journal of Applied Psychology, Academy of Management Journal, Journal of Management, and Personnel Psychology. We investigated the extent to which sample characteristics and response rates were reported in articles among these top journals—extending prior work by Shen et al. (Reference Shen, Kiger, Davies, Rasch, Simon and Ones2011). Research in these journals during this timeframe reported on 345 survey samples. Many articles included multiple samples, and these were coded as separate occasions rather than combined into one code. Out of the 345 survey samples, only 32 (9.3%) were representative of either the organization studied or drawn from or the target population at large. Representativeness was coded liberally as a “Yes” if the author(s) indicated that the sample was representative of the organization from which the author(s) obtained the sample or the population at large. This finding suggests that researchers either do not obtain representative samples or do not deem it important enough to mention so in their manuscripts. This finding is particular troubling because, as Landers and Behrend asserted, organizational samples are subject to a host of idiosyncrasies that may not be present across the target population (e.g., organizational culture, industry, selection instruments used). If a sample used within a particular organization is not representative of that organization—much less the target population—it is possible that still more sample idiosyncrasies may affect the external validity of a study's findings (Hanges & Wang, Reference Hanges, Wang and Kozlowski2012).

Our literature review also sought to investigate response rates from survey research studies. The average response rate across all 345 coded samples was 61.02%, a response rate that is well above the average for organizational research (53%; Baruch & Holtom, Reference Baruch and Holtom2008). However, out of the 345 survey samples coded, 105 (30.43%) were missing response rate information. This oversight is problematic because researchers do not consistently provide important information to help readers assess the quality of the sampling used in the research. If response rate information is missing, then it is also likely that results of a nonresponse analysis to assess nonresponse bias are not included either.

Last, we assert that little attention is paid to sampling among I-O psychologists because it is not emphasized during graduate training at the onset of their careers. Current graduate training in I-O psychology typically requires a single course on research methodology, often taught to psychology graduate students across different areas within psychology in the same course. Such a course likely provides an overview of the various types of sampling, their pros and cons, as well as the frequency with which they occur in psychological research. This information is usually reiterated to some extent in statistics and measurement courses also taken at the graduate level. Despite the reoccurrence of this topic, most teaching material rarely goes beyond the sort of dichotomous description discussed in the focal article—that is, sampling strategies are usually broadly painted as “good” or “bad.” In stark contrast, measurement issues are studied much more deeply in graduate level coursework in I-O psychology (Byrne et al., Reference Byrne, Hayes, McPhail, Hakel, Cortina and McHenry2014). Therefore, measurement issues rise to the forefront, whereas sampling strategy takes a back seat. Students may not learn to recognize its importance and consequently apply a limited focus on sampling in their work in science and/or practice. In contrast to psychology, other fields, such as sociology, economics, epidemiology, and survey methodology, place more emphasis on sampling error as an important source of measurement error (e.g., Biemer, Groves, Lyberg, Mathiowetz, & Sudman, Reference Biemer, Groves, Lyberg, Mathiowetz and Sudman2011).

Does Sampling Matter?

The next question is whether and to what degree sampling really matters in research studies. Grzywacz, Carlson, and Reboussin (Reference Grzywacz, Carlson, Reboussin, Grzywacz and Demerouti2013) conducted simulations to examine the effect of overrepresentations of women in samples of studies of work/family spillover. Their simulations demonstrated that sampling can impact the results of a research study. Specifically, they found that differences in proportions of males and females in a research study could lead to different results regarding work/family experiences, highlighting the importance of sample composition.

I-O psychologists traditionally rely on meta-analysis as a tool for estimating “true” relations among variables, correcting for measurement error, and averaging across methodological differences among studies. Results by Grzywacz et al. (Reference Grzywacz, Carlson, Reboussin, Grzywacz and Demerouti2013) remind us that outcomes of a meta-analysis may depend on the sampling used in studies included in the meta-analysis. Therefore it is critical to assess the impact of sample characteristics as well as adjust for sampling error when conducting meta-analyses (Schmidt & Hunter, Reference Schmidt and Hunter2014).

Recommendations for Sampling in I-O Psychology Research

We do not suggest focusing less on measurement but rather advocate that I-O psychologists pay more attention to the issue of properly sampling research participants to increase external validity and be clearer about sample composition. In particular, I-O psychologists need to develop clear criteria regarding what constitutes “good” or even “acceptable” sampling.

Next, I-O psychologists should focus more effort on the specification of the sampling frame for a particular study. This step follows after identifying the population of interest; the sample frame includes the methods by which all elements of the target population will be obtained. The goal is to obtain as much coverage of the target population as possible within a particular sample frame (Groves & Couper, Reference Groves, Couper, Ploeg, Moffitt and Citro2002; Groves et al., Reference Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau2009). We suggest that sample frame specification is often overlooked in I-O psychology research. As Landers and Behrend pointed out, we rely on the “best available” in a convenience sample rather than devote considerably more effort to designing the sampling frame for a study. Sampling may include deriving a probability sample (i.e., members or elements of a sample frame are selected using chance methods), stratified sampling (classifying the sample into subpopulations based on supplementary information, e.g., geographical location or individuals’ demographic characteristics, such as age, race, or gender) and then selecting sample members separately for each stratum, proportionate stratification (sample elements are selected from each stratum in proportion to the stratum's frequency in the population), disproportionate sampling (selecting units from the population at a different rate than they occur in the population), clustered sampling (a sampling technique in which frame elements are selected jointly rather than individually; Groves et al., Reference Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau2009), or various types of nonprobability sampling (convenience, snowball, etc.). In I-O psychology we should accurately identify the type of sampling strategy used and provide a proper justification for such methods when describing the research design.

Next, we call for researchers to continue to examine evidence of sampling error, which occurs when not all elements of a sampling frame are measured. Typically in I-O psychology we assess sampling error by examining response rates to surveys. Results of our literature review described above suggest room for improvement. Response rates should be reported, nonresponse should be systematically investigated, and one should not assume that large samples or samples obtained with a high response are automatically representative of the target population. It is possible that even when the response rate is high, nonresponse may occur disproportionately in relation to key characteristics of sample members (Rogelberg & Stanton, Reference Rogelberg and Stanton2007). Thus, missing data analyses to systematically investigate nonresponse become critically important. As per the recommendation by Rogelberg and Stanton (Reference Rogelberg and Stanton2007), journals should offer clear instructions to authors regarding the reporting of survey response rates as well as regarding how authors ought to address nonresponse in their research.

Last, we call for more emphasis on sampling in research methods training in graduate I-O psychology programs. The next generation of I-O psychologists should understand and attend more to sampling issues in research, including the use of proper methods (e.g., focus on the sample frame and assessment of sampling error) as described above and reporting of procedures. Training may be more effective in I-O psychology if it addresses particular issues or challenges specific to organizational research rather than training students about sampling in the same manner across all areas of psychology.