The purpose of our commentary is to expand the conversation started by the focal article (Bergman & Jean, Reference Bergman and Jean2016) by providing a practitioner perspective, especially focusing on practitioners who work in a government or military context. We believe this perspective is valuable because (a) interventions and research projects involving nonprofessional/nonmanagement workers are already regularly being conducted in this context; (b) there are likely to be other causes of a lack of focus on broader populations of workers in this context than those highlighted by the focal article, some of which may differ between practitioners and academics; and (c) we believe practitioners in this context have much to contribute to solving the potential problems that the focal article highlights.
Practitioners Already Interact With a Broad Sample of Workers
The focal article notes that applied work with undersampled populations is almost certainly already being done in many organizations, even if unpublished. We agree and will present below some suggestions for why this work is not published. However, we speculate that if the focal article's analyses of journal articles had been expanded to a larger time frame, more examples of research involving workers may have emerged. In the service of looking forward, we highlight several high-impact studies in the industrial–organizational (I-O) literature that have sampled across many organizational levels in government and military contexts, also noting where current work is being done.
-
• Project A purposely sampled across occupations within the U.S. Army and spawned a host of publications and books/book chapters (e.g., Campbell & Knapp, Reference Campbell and Knapp2001). Jobs studied and reported on as part of Project A included military-specific jobs such as infantry but also included jobs with substantial nonmilitary duties, such as engineers, mechanics, food service workers, medical specialists, and carpentry/masonry specialists. The U.S. Army still conducts personnel research using purposive sampling across/within the range of needed jobs, including the development of measures of personality (Stark et al., Reference Stark, Chernyshenko, Drasgow, Nye, White, Heffner and Farmer2014), information technology knowledge (Trippe, Moriarty, Russell, Carretta, & Beatty, Reference Trippe, Moriarty, Russell, Carretta and Beatty2014), and work interests (Ingerick & Rumsey, Reference Ingerick and Rumsey2014). Notably, a recent National Academy of Sciences report outlined a future basic science research agenda that could be conducted in partnership with the U.S. Army (National Research Council, 2015) and would likely have implications for non-Army jobs filled by workers.
-
• Research supporting the General Aptitude Test Battery (GATB) was conducted by the U.S. Department of Labor (DOL) and U.S. Office of Personnel Management (OPM) and sampled across 515 validity studies for jobs ranging across the complexity spectrum, including many in jobs underrepresented by the current literature. This research led to what the focal article refers to as the law of general mental ability and performance (e.g., Hunter & Hunter, Reference Hunter and Hunter1984). Other work using this broadly sampled database provided foundations for test bias research (Hartigan & Wigdor, Reference Hartigan and Wigdor1989). Today, psychologists in federal agencies continue to study the use of employment assessments across many jobs, whether for high-stakes (selection) or low-stakes (development; analytics) purposes.
-
• The U.S. Department of Labor's Occupational Information Network (O*NET) provides job-analytic information for occupations representing jobs across the U.S. economy. Data from O*NET have been used to study topics as diverse as occupational literacy requirements (LaPolice, Carter, & Johnson, Reference LaPolice, Carter and Johnson2008), managerial role theory (Dierdorff, Rubin, & Morgeson, Reference Dierdorff, Rubin and Morgeson2009), and expatriate effectiveness (Shin, Morgeson, & Campion, Reference Shin, Morgeson and Campion2007). Sackett and Walmsley (Reference Sackett and Walmsley2014) used O*NET data in a study of occupational personality requirements, exactly for the purposes noted in the focal article: addressing drawbacks of other sampling strategies that undersample workers relative to their prevalence in Bureau of Labor Statistics research. Currently, O*NET data are updated with regular frequency according to a sampling plan, which involves contact with workers across many occupations.
We understand the focal authors’ primary point was that work with these sorts of samples is not being proportionately published in our field's journals. We argue that if work with at least some of these types of samples is already being done, there's a relatively much smaller barrier to publishing it. First, there may be opportunities to publish current work or ongoing projects. Second, and perhaps more important, if we currently have established access to workers, we may be able to plan future studies with the goal of sampling workers. We expand on these points below, noting both obstacles to and prospects for including workers in applied studies.
Sampling Constraints for Practitioners
In spite of the above examples, practitioner-sourced research may also potentially be biased toward overrepresented populations. The focal article highlights a number of potential causes for why there may be an overreliance on management and professional samples in the published literature. We agree that many of them are plausible. However, we also argue that there are likely to be additional factors at play among practitioners. Among many potential factors, we highlight several in this section that we believe are central to the issue.
First, much of the work practitioners do is not based on the drawing of the most readily available samples but rather on particular samples of interest from sponsors. In other words, the practice of I-O usually has specific targeted population(s) in mind (e.g., “I want to understand work environment perceptions among administrative staff” or “I need to develop a selection system for special agents”). Practitioners are often contracted to work with only one specific segment of a sponsoring organization, and thereby the questions of interest are often different from those appearing in the literature. The extent to which this biases the focus of research toward management/professional workers speaks to the organizational biases the focal article highlighted and is an area where academics and practitioners have similar constraints. However, academics have the freedom to search out willing organizations and trade services for research access. Practitioners, on the other hand, are generally exposed to samples as a consequence of contracting for work with clients/sponsors or as a function of the types of organizations that hire internal I-O psychologists. Examples include federal agencies that retain I-O psychologists to focus specifically on “mission critical” occupations, such as law enforcement or military personnel. In contrast, the workers described in the focal article may tend to hold “mission support” jobs. These forces may operate for or against contact with workers.
Moreover, practitioners are ethically bound to recommend and implement interventions that are going to be of value/benefit to their client organization. Although research investigating secondary research questions can often be added with the sponsor's permission, the overriding aim of most applied research programs is to implement an intervention for which there is evidence of efficacy. The focal article mentions several topics that may undersample workers, such as holding multiple jobs, job flexibility/autonomy, and economic tenuousness. Research on these topics may be viewed as having an uncertain link to organizational objectives, or these topics may simply be regarded as one of many contributing factors in explaining differences in job performance, satisfaction, or other dependent variables. Studies geared toward basic science objectives may be acceptable but will usually be of secondary importance to the immediate needs of the applied research, even in government or military settings. This means that practitioner research may be likely to be restricted to interventions (broadly defined) where I-O psychologists already play a role, that affect all or large numbers of employees (e.g., recruiting, selection, training/onboarding, performance appraisal), and/or where relevant variables are easily measured (e.g., employee attitude surveys). It may be possible to push organizations toward new areas of research, but an uncertain link to organizational objectives can make this a tough sell.
In addition, there are features of the publishing process itself that may not overlap well with applied work. Research generally needs to fit within the current zeitgeist to be publishable in many of our field's journals, suggesting at least four considerations (to be simplistic): (a) whether the research question is viewed as being of interest to an outlet's readership, (b) whether the narrative framing and findings of the research are judged as sufficiently couched in theory, (c) whether research is judged to be of sufficient novelty, and (d) whether sufficient methodological and analytical rigor are applied to answer the research question or reach an outcome. We suggest that these publishing-focused considerations are often incongruent with the application purposes focused on by organizations retaining I-O psychologists. A sponsoring organization may have no particular appreciation for or interest in a, b, or c, with the main emphasis on d.
Finally, scholarship is a primary role of most academics’ jobs. On the other hand, many practitioners are in situations where their employers appreciate publishing but will provide minimal or no support for doing so. In addition, incentives and rewards (e.g., raises, promotions) are usually provided on the basis of activities more explicitly tied to organizational performance. This may reduce the motivation for many practitioners to involve themselves in the publishing process, given the time investment involved in writing a manuscript, responding to revisions, and potentially doing so at multiple outlets. To the extent that this means that data from potentially unique samples are never published, this could contribute to the problem highlighted by the focal article.
In addition to the constraints we have described, a reviewer raised the possibility of publication bias stemming from practitioner pressures not to publish negative results. Given the goal of increasing representation and sampling of workers in the literature and recognizing that negative results are already difficult to publish in journals, we anticipate that although publication bias is possible for replication or boundary condition studies, it would likely be no greater of an issue than with the current literature, and the benefits would outweigh any drawbacks.
The Role of Practitioners in Expanding the Scope of Our Science
To the extent that research and interventions among broader populations of workers is already being conducted, much progress can potentially be made on this problem in a relatively short amount of time. It is hoped that this set of articles will be a useful first step and inspire any practitioners with data that speak to the boundary conditions of the theories we use in I-O psychology to publish their findings. For instance, previous studies may have been conducted specifically focused on undersampled populations or databases may have been compiled across all sorts of jobs from which subsets can be extracted (e.g., O*NET or other government surveys). Even within the constraints of applied research, this research can provide useful replications and reach broader samples consistent with the focal article's call. By drawing attention to this issue, the likelihood of successful publication could be improved. This process could be enhanced by collaboration with leading scholars in relevant topic areas, and this may also help combat the barrier of a lack of practitioner incentives for publishing. For instance, practitioners may be on the front line of identifying emerging problems that would expand our literature; collaboration with academics on these emerging issues is a good strategy for planning research in ways that include worker samples and illuminate important issues. Indeed, as with many problems our field faces, the interplay between scholars and practitioners is an important part of the solution.
The focal article largely focuses on action steps among I-O psychologists as solutions (e.g., changing journal policy, looking in different places for samples, and being open to qualitative work). However, in terms of influencing future applied research, one of the most important steps for closing this gap may be to attempt to educate and/or persuade sponsor organizations of the importance of novel research questions, whether that means researching the effects of extending interventions to understudied populations or investigating a type of question in which the organization hasn't typically been interested. If a case can be made that a research program can help contribute to organizational effectiveness or understanding how the organization operates, it is probably much more likely to be implemented than an appeal to basic science. The first step of publishing research from more representative samples is getting in the door to be able to conduct the studies in the first place.
General adoption of the mindset that any given applied project may have publishable elements can help practitioners plan ahead in this regard. This approach can encourage identification of actionable research questions, careful specification of populations to which research questions pertain (which beget sampling strategies), plans for working with organizations to measure the necessary variables (which may extend well beyond using different methods for administering questionnaires), and consideration of whether collaborations may be feasible. Practitioner planning and responsibility for conducting sound research in constrained environments is one side of the issue; journals also may need to revisit implied policies to ensure that high-quality applied research and its associated constraints are given full consideration in the review process. That is, recognition that useful studies may not have started with identification of a targeted outlet (i.e., consideration of a journal's readership interests, relative emphasis on theoretical development or novelty) could play a role in the evaluation of manuscripts. For instance, it may be very costly to add a single item, let alone a scale, to a global employee survey, or it may be difficult to convince a sponsoring organization to allow the type of experimental designs or data collection strategies that pose potential drains on resources (e.g., hourly employee time). One I-O–focused journal has recently been conceived in response to some of these issues: Personnel Assessment and Decisions, which focuses on brief applied articles of practical relevance, along with practitioner demonstrations of applying novel solutions to organizational measurement-focused problems. Nothing should change in terms of acceptance of only high-quality work, but the defining characteristics of high quality may need to incorporate recognition of these constraints to be more inclusive of contributions that practitioners, whether in government, military, or industry, can make.
The purpose of our commentary is to expand the conversation started by the focal article (Bergman & Jean, Reference Bergman and Jean2016) by providing a practitioner perspective, especially focusing on practitioners who work in a government or military context. We believe this perspective is valuable because (a) interventions and research projects involving nonprofessional/nonmanagement workers are already regularly being conducted in this context; (b) there are likely to be other causes of a lack of focus on broader populations of workers in this context than those highlighted by the focal article, some of which may differ between practitioners and academics; and (c) we believe practitioners in this context have much to contribute to solving the potential problems that the focal article highlights.
Practitioners Already Interact With a Broad Sample of Workers
The focal article notes that applied work with undersampled populations is almost certainly already being done in many organizations, even if unpublished. We agree and will present below some suggestions for why this work is not published. However, we speculate that if the focal article's analyses of journal articles had been expanded to a larger time frame, more examples of research involving workers may have emerged. In the service of looking forward, we highlight several high-impact studies in the industrial–organizational (I-O) literature that have sampled across many organizational levels in government and military contexts, also noting where current work is being done.
• Project A purposely sampled across occupations within the U.S. Army and spawned a host of publications and books/book chapters (e.g., Campbell & Knapp, Reference Campbell and Knapp2001). Jobs studied and reported on as part of Project A included military-specific jobs such as infantry but also included jobs with substantial nonmilitary duties, such as engineers, mechanics, food service workers, medical specialists, and carpentry/masonry specialists. The U.S. Army still conducts personnel research using purposive sampling across/within the range of needed jobs, including the development of measures of personality (Stark et al., Reference Stark, Chernyshenko, Drasgow, Nye, White, Heffner and Farmer2014), information technology knowledge (Trippe, Moriarty, Russell, Carretta, & Beatty, Reference Trippe, Moriarty, Russell, Carretta and Beatty2014), and work interests (Ingerick & Rumsey, Reference Ingerick and Rumsey2014). Notably, a recent National Academy of Sciences report outlined a future basic science research agenda that could be conducted in partnership with the U.S. Army (National Research Council, 2015) and would likely have implications for non-Army jobs filled by workers.
• Research supporting the General Aptitude Test Battery (GATB) was conducted by the U.S. Department of Labor (DOL) and U.S. Office of Personnel Management (OPM) and sampled across 515 validity studies for jobs ranging across the complexity spectrum, including many in jobs underrepresented by the current literature. This research led to what the focal article refers to as the law of general mental ability and performance (e.g., Hunter & Hunter, Reference Hunter and Hunter1984). Other work using this broadly sampled database provided foundations for test bias research (Hartigan & Wigdor, Reference Hartigan and Wigdor1989). Today, psychologists in federal agencies continue to study the use of employment assessments across many jobs, whether for high-stakes (selection) or low-stakes (development; analytics) purposes.
• The U.S. Department of Labor's Occupational Information Network (O*NET) provides job-analytic information for occupations representing jobs across the U.S. economy. Data from O*NET have been used to study topics as diverse as occupational literacy requirements (LaPolice, Carter, & Johnson, Reference LaPolice, Carter and Johnson2008), managerial role theory (Dierdorff, Rubin, & Morgeson, Reference Dierdorff, Rubin and Morgeson2009), and expatriate effectiveness (Shin, Morgeson, & Campion, Reference Shin, Morgeson and Campion2007). Sackett and Walmsley (Reference Sackett and Walmsley2014) used O*NET data in a study of occupational personality requirements, exactly for the purposes noted in the focal article: addressing drawbacks of other sampling strategies that undersample workers relative to their prevalence in Bureau of Labor Statistics research. Currently, O*NET data are updated with regular frequency according to a sampling plan, which involves contact with workers across many occupations.
We understand the focal authors’ primary point was that work with these sorts of samples is not being proportionately published in our field's journals. We argue that if work with at least some of these types of samples is already being done, there's a relatively much smaller barrier to publishing it. First, there may be opportunities to publish current work or ongoing projects. Second, and perhaps more important, if we currently have established access to workers, we may be able to plan future studies with the goal of sampling workers. We expand on these points below, noting both obstacles to and prospects for including workers in applied studies.
Sampling Constraints for Practitioners
In spite of the above examples, practitioner-sourced research may also potentially be biased toward overrepresented populations. The focal article highlights a number of potential causes for why there may be an overreliance on management and professional samples in the published literature. We agree that many of them are plausible. However, we also argue that there are likely to be additional factors at play among practitioners. Among many potential factors, we highlight several in this section that we believe are central to the issue.
First, much of the work practitioners do is not based on the drawing of the most readily available samples but rather on particular samples of interest from sponsors. In other words, the practice of I-O usually has specific targeted population(s) in mind (e.g., “I want to understand work environment perceptions among administrative staff” or “I need to develop a selection system for special agents”). Practitioners are often contracted to work with only one specific segment of a sponsoring organization, and thereby the questions of interest are often different from those appearing in the literature. The extent to which this biases the focus of research toward management/professional workers speaks to the organizational biases the focal article highlighted and is an area where academics and practitioners have similar constraints. However, academics have the freedom to search out willing organizations and trade services for research access. Practitioners, on the other hand, are generally exposed to samples as a consequence of contracting for work with clients/sponsors or as a function of the types of organizations that hire internal I-O psychologists. Examples include federal agencies that retain I-O psychologists to focus specifically on “mission critical” occupations, such as law enforcement or military personnel. In contrast, the workers described in the focal article may tend to hold “mission support” jobs. These forces may operate for or against contact with workers.
Moreover, practitioners are ethically bound to recommend and implement interventions that are going to be of value/benefit to their client organization. Although research investigating secondary research questions can often be added with the sponsor's permission, the overriding aim of most applied research programs is to implement an intervention for which there is evidence of efficacy. The focal article mentions several topics that may undersample workers, such as holding multiple jobs, job flexibility/autonomy, and economic tenuousness. Research on these topics may be viewed as having an uncertain link to organizational objectives, or these topics may simply be regarded as one of many contributing factors in explaining differences in job performance, satisfaction, or other dependent variables. Studies geared toward basic science objectives may be acceptable but will usually be of secondary importance to the immediate needs of the applied research, even in government or military settings. This means that practitioner research may be likely to be restricted to interventions (broadly defined) where I-O psychologists already play a role, that affect all or large numbers of employees (e.g., recruiting, selection, training/onboarding, performance appraisal), and/or where relevant variables are easily measured (e.g., employee attitude surveys). It may be possible to push organizations toward new areas of research, but an uncertain link to organizational objectives can make this a tough sell.
In addition, there are features of the publishing process itself that may not overlap well with applied work. Research generally needs to fit within the current zeitgeist to be publishable in many of our field's journals, suggesting at least four considerations (to be simplistic): (a) whether the research question is viewed as being of interest to an outlet's readership, (b) whether the narrative framing and findings of the research are judged as sufficiently couched in theory, (c) whether research is judged to be of sufficient novelty, and (d) whether sufficient methodological and analytical rigor are applied to answer the research question or reach an outcome. We suggest that these publishing-focused considerations are often incongruent with the application purposes focused on by organizations retaining I-O psychologists. A sponsoring organization may have no particular appreciation for or interest in a, b, or c, with the main emphasis on d.
Finally, scholarship is a primary role of most academics’ jobs. On the other hand, many practitioners are in situations where their employers appreciate publishing but will provide minimal or no support for doing so. In addition, incentives and rewards (e.g., raises, promotions) are usually provided on the basis of activities more explicitly tied to organizational performance. This may reduce the motivation for many practitioners to involve themselves in the publishing process, given the time investment involved in writing a manuscript, responding to revisions, and potentially doing so at multiple outlets. To the extent that this means that data from potentially unique samples are never published, this could contribute to the problem highlighted by the focal article.
In addition to the constraints we have described, a reviewer raised the possibility of publication bias stemming from practitioner pressures not to publish negative results. Given the goal of increasing representation and sampling of workers in the literature and recognizing that negative results are already difficult to publish in journals, we anticipate that although publication bias is possible for replication or boundary condition studies, it would likely be no greater of an issue than with the current literature, and the benefits would outweigh any drawbacks.
The Role of Practitioners in Expanding the Scope of Our Science
To the extent that research and interventions among broader populations of workers is already being conducted, much progress can potentially be made on this problem in a relatively short amount of time. It is hoped that this set of articles will be a useful first step and inspire any practitioners with data that speak to the boundary conditions of the theories we use in I-O psychology to publish their findings. For instance, previous studies may have been conducted specifically focused on undersampled populations or databases may have been compiled across all sorts of jobs from which subsets can be extracted (e.g., O*NET or other government surveys). Even within the constraints of applied research, this research can provide useful replications and reach broader samples consistent with the focal article's call. By drawing attention to this issue, the likelihood of successful publication could be improved. This process could be enhanced by collaboration with leading scholars in relevant topic areas, and this may also help combat the barrier of a lack of practitioner incentives for publishing. For instance, practitioners may be on the front line of identifying emerging problems that would expand our literature; collaboration with academics on these emerging issues is a good strategy for planning research in ways that include worker samples and illuminate important issues. Indeed, as with many problems our field faces, the interplay between scholars and practitioners is an important part of the solution.
The focal article largely focuses on action steps among I-O psychologists as solutions (e.g., changing journal policy, looking in different places for samples, and being open to qualitative work). However, in terms of influencing future applied research, one of the most important steps for closing this gap may be to attempt to educate and/or persuade sponsor organizations of the importance of novel research questions, whether that means researching the effects of extending interventions to understudied populations or investigating a type of question in which the organization hasn't typically been interested. If a case can be made that a research program can help contribute to organizational effectiveness or understanding how the organization operates, it is probably much more likely to be implemented than an appeal to basic science. The first step of publishing research from more representative samples is getting in the door to be able to conduct the studies in the first place.
General adoption of the mindset that any given applied project may have publishable elements can help practitioners plan ahead in this regard. This approach can encourage identification of actionable research questions, careful specification of populations to which research questions pertain (which beget sampling strategies), plans for working with organizations to measure the necessary variables (which may extend well beyond using different methods for administering questionnaires), and consideration of whether collaborations may be feasible. Practitioner planning and responsibility for conducting sound research in constrained environments is one side of the issue; journals also may need to revisit implied policies to ensure that high-quality applied research and its associated constraints are given full consideration in the review process. That is, recognition that useful studies may not have started with identification of a targeted outlet (i.e., consideration of a journal's readership interests, relative emphasis on theoretical development or novelty) could play a role in the evaluation of manuscripts. For instance, it may be very costly to add a single item, let alone a scale, to a global employee survey, or it may be difficult to convince a sponsoring organization to allow the type of experimental designs or data collection strategies that pose potential drains on resources (e.g., hourly employee time). One I-O–focused journal has recently been conceived in response to some of these issues: Personnel Assessment and Decisions, which focuses on brief applied articles of practical relevance, along with practitioner demonstrations of applying novel solutions to organizational measurement-focused problems. Nothing should change in terms of acceptance of only high-quality work, but the defining characteristics of high quality may need to incorporate recognition of these constraints to be more inclusive of contributions that practitioners, whether in government, military, or industry, can make.