Study Goals and Hypotheses Dictate Sample Adequacy

An important foundation for this commentary is to note that the study purpose and hypotheses affect choice of samples, as sample adequacy is judged on representativeness of the population in which research results are applied (Guion, Reference Guion and Rogelberg2002). Therefore, it is important to articulate when the underrepresentation of workers in applied psychology literature would influence the advancement of science, as I-O researchers often appropriately and inappropriately use worker/nonworker, student, nonworker, and worker samples. Indeed, reviewing the articles summarized by Bergman and Jean (i.e., articles published in Journal of Applied Psychology, Personnel Psychology, Academy of Management Journal, Journal of Organizational Behavior, and Journal of Management during 2012–2014) reveals many appropriate examples in each sample category.

Arguably, some of the most difficult samples to justify in applied psychology are workers and nonworkers included in a study. Yet, under many circumstances, these may be the most appropriate samples. One such example is the work of Wang, Sui, Luthans, Wang, and Wu (Reference Wang, Sui, Luthans, Wang and Wu2014), which investigated leadership effects on employee performance with a sample of leaders and their followers. Not only is this an excellent example of correctly using a sample of workers/nonworkers to address study goals and hypotheses, but it adheres to Bergman and Jean's assertion (see also Meindl, Reference Meindl and Munighan1993) that workers need to be examined even when research is addressing leadership system questions (p. 104). Similarly, the advocacy for student samples can be provided by the work of Le, Robbins, and Westrick (Reference Le, Robbins and Westrick2014), in which data from students addressed the study goals and hypotheses revolving around student choice and persistence in the science, technology, engineering, and mathematics (STEM) fields. Likewise, Scott, Garza, Conlon, and Kim (Reference Scott, Garza, Conlon and Kim2014) conducted a study investigating managerial motives for adhering to justice rules on a day-to-day basis. Using daily surveys of managers, the authors were able to address their study goals and hypotheses with an appropriate nonworker sample. Finally, Qin, Direnzo, Xu, and Duan (Reference Qin, DiRenzo, Xu and Duan2014) published an excellent example of the use of worker samples. In two studies assessing relationships between emotional exhaustion and prohibitive voice behavior, the authors used two independent samples of construction workers (e.g., welders, carpenters, bricklayers) that clearly matched the study goals and hypotheses.

In sum, we believe there are many exemplary examples of studies utilizing worker/nonworker, student, nonworker, and worker samples within the applied psychology literature. Unfortunately, even when samples are used appropriately, yet these samples at the aggregate do not reflect the labor market, the researcher bias toward managers, professionals, and executives mentioned by Bergman and Jean is present (i.e., it is not the samples but the research that is biased). Thus, even in the best case scenario (i.e., appropriate sample use), this is a pervasive problem that ultimately affects our current and future body of theoretical and empirical knowledge. Equally problematic is the inappropriate use of worker/nonworker, student, nonworker, and worker samples, which may explain why many constructs and phenomena may not be fully understood, as in these cases sample adequacy is ignored.

Different Applied Disciplines Require Different Samples

To get a clearer picture of the underrepresentation of appropriate worker samples in I-O psychology, we replicated the results of Bergman and Jean's focal article with more specialized journals that should arguably include more worker samples. That is, each of the journals outlined in the focal article broadly represents the applied psychology literature and provides a benchmark for applied research across subdisciplines (e.g., career development; leadership; team development, processes, and effectiveness; work stress, health, and well-being). One area that we feel should use more worker samples is the field of OHP, as OHP is generally concerned with protecting and promoting the safety, health, and well-being of workers by focusing on the work environment, the individual, and the work–family interface (cf. OHP journal descriptions). The primary journals for OHP are Journal of Occupational Health Psychology (JOHP) and Work & Stress (W&S); thus we coded these journals based on the coding scheme presented by Bergman and Jean for the years 2012–2014.

Given the specific scope of JOHP and W&S on health, safety, and well-being, we anticipated results closer to the labor market data outlined by Bergman and Jean (i.e., greater focus on workers). Based on JOHP and W&S data (see Figure 1), it seems that OHP journals publish a much larger proportion of worker samples than do more general applied psychology journals, as workers consisted of the second most frequent category versus the lowest frequency in the review by Bergman and Jean. Despite the large portion of samples focusing on workers, these results still do not reflect the true labor market and underrepresent workers, as only 18.3% and 33.8% of the samples in JOHP and W&S were worker samples. Thus, even in this better case scenario (i.e., greater representation of worker samples), appropriate worker samples still appear to be underrepresented in a field that should focus largely on workers.

Figure 1.

Proportion of samples from top occupational health psychology journals (2012–2014), classified by worker status. JOHP = Journal of Occupational Health Psychology; W&S = Work & Stress.