In their focal article, Schneider and Pulakos (Reference Schneider and Pulakos2022) review research findings on organizational (vs. individual) differences in several areas of industrial-organizational (I-O) psychology and discuss future research agendas and avenues. In particular, they note a paucity of organization-level research in personnel selection, stating that “relatively little research has examined the effect of selection practices on organizational performance” (p. X) and that “there is scant evidence that organizations that use our selection methods outperform those that do not” (p. X). We concur, but we discuss additional relevant research findings and their implications that shed further light on organizational differences in personnel selection practices/procedures and their influence on organizational performance in this commentary. In doing so, we hope this commentary will encourage I-O researchers to conduct more research on organizational differences in personnel selection using the same methodological rigor we have successfully used in abundant research on individual differences. We hope that this commentary will be helpful to I-O practitioners in who are charge of designing and evaluating organization-level selection practices and systems.
Many I-O psychologists often hear that their research findings, despite sophisticated in terms of study design and analysis, are too micro-level-oriented; in other words, they primarily focus on individual job performance when discussing selection procedures/practices that also have macro-level implications (e.g., organizational performance) with organizational decision makers. For example, human resource (HR) managers trained in I-O psychology are well aware that some personnel selection procedures are more positively related to individual job performance than are others, and thus they desire to administer the most valid selection procedure if all other things are equal. However, when they pitch the idea to their organization’s upper-level managers, they often encounter a question of whether organizations that use such a selection procedure (more selectively) would outperform others that do not use it (or use it less selectively). Of course, those HR managers can mention selection utility and its formula. However, utility analysis attempts to estimate the financial value of a selection procedure based on individual-level validity, thereby possibly making a “bottom-up” inferential leap (Schneider etal., Reference Schneider, Smith, Sipe, Klein and Kozlowski2000). As pointed out in Schneider and Pulakos (Reference Schneider and Pulakos2022), whereas utility analysis can explain within-organization changes/differences in performance (e.g., would organizations be more profitable than before if they used a more valid selection procedure?), it cannot account for between-organizations differences in performance (e.g., would organizations that use a more valid selection procedure be more profitable than those that do that use it?). That is, HR managers often struggle to answer the aforementioned question given no relevant I-O research findings.
In the strategic/macro HR management (SHRM) literature, it is well known that organizations that more carefully hire their employees outperform others that do not. Combs etal. (Reference Combs, Liu, Hall and Ketchen2006) provided meta-analytic evidence that organization-level selection selectivity is positively correlated with organizational performance (mean r = .11; k = 15; 95% CI = [.08, .14]; corrected r = .14); for comparison, it is noted that pay for performance (incentive compensation) is almost identically correlated with organizational performance (mean r = .11; k = 31; 95% CI = [.08, .15]; corrected mean r = .15), and performance appraisal has a much lower correlation with organizational performance (mean r = .03; k = 8; 95% CI = [-.05, .12]; corrected mean r = .04). The aforementioned findings can be useful to both HR managers and organizational decision makers, and they may want to invest more in personnel selection and incentive compensation practices than in performance appraisal practices. Unfortunately, this kind of insight is often unavailable to many I-O researchers and practitioners. However, it should be noted that the above-mentioned findings are based on several line, general, and/or HR managers’ responses to several questions of whether or how intensively such HR practices are implemented (e.g., the degree to which the organization hires employees rigorously and selectively). In the eyes of I-O researchers, this approach is likely to be seen as suboptimal; such reports are likely to suffer from measurement error and response distortion (Gerhart, Reference Gerhart2012). Concerning the main objective of this commentary (organizational differences in personnel selection), the broad questions used in this approach provide few nuanced insights into how an organization should design a personnel selection system to compete better with other organizations. In particular, it is difficult to know which selection procedures should be prioritized to maximize organizational (vs. individual) performance. Below, we discuss several studies that we think can address these limitations.
Terpstra and Rozell (Reference Terpstra and Rozell1993, n = 201 U.S. organizations) found that the use (vs. nonuse) of (a) various types of information in application blanks, (b) local validation studies, and (c) cognitive ability tests is positively correlated with overall organizational performance (r = .17, .13, and .08, respectively). However, they found that the use of structured interviews was virtually uncorrelated with overall organizational performance (r = -.01), which may appear quite counterintuitive to those with “the individual mindset” given that structured interviews are one of the most valid predictors of individual job performance (Schmidt & Hunter, Reference Schmidt and Hunter1998). The use of cognitive ability tests was found to be less positively related to overall organizational performance than the use of application blanks, which may also be viewed as counterintuitive because cognitive ability tests are more valid than application blanks at the individual level. In the eyes of I-O researchers, a major limitation of Terpstra and Rozell’s study is that both selection procedures and organizational performance were measured using the same source (mostly HR department heads or executives) on the same survey at the same time. Furthermore, the surveyed organizations were limited to those with 200 or more employees included in Dun’s Business Rankings (1989). Thus, their findings were subject to both common method bias and self-selection bias (range restriction). Relatedly, Drogan and Yancey (Reference Drogan and Yancey2011; n = 122 credit unions in the US) failed to replicate Terpstra and Rozell’s findings in the financial services industry when they used objective (vs. subjective) measures of organizational performance. Astute I-O researchers would critique that their findings might not generalize to other industries. In summary, we believe Terpstra and Rozell’s study provided an initial yet important data-based insight that individual-level findings in personnel selection might not be generalizable and identical to the organizational level.
More recently, Kim and Ployhart (Reference Kim and Ployhart2018) examined selection practice usage and its influence on objective measures of organizational performance. Although it was not reported in the original paper, an additional analysis (see our Table 1) of the data used in Kim and Ployhart’s study (n = 409–411 organizations in South Korea) has revealed that the use (vs. nonuse) of only 4 of the 12 selection procedures examined—that is, group discussion interviews (r = .31, .32), personality tests (r = .22, .23), job aptitude tests (r = .17, .18), and professional ability interviews (r = both .11)—have significant (sizable) correlations with operational (operating profit per employeeFootnote 1) and financial organizational performance (earnings before interest and taxes [EBIT]). The others—professional ability tests (r = both .01), general intelligence tests (r = .01, .00), general interviews (r = .00, .01), assessment centers (r = both .08), practical (work sample) tests (r = -.02, -.03), recommendation (r = -.04, -.05), statement of purpose (r = both .05), and internship (r = -.02, -.01)—are not significantly correlated with operational and financial organizational performance. Notably, the use of group discussion interviews and personality tests is more positively correlated with organizational performance than the use of other (more cognitively loaded) selection procedures, such as general intelligence tests and assessment centers (Schmidt & Hunter, Reference Schmidt and Hunter1998). This may be counterintuitive to I-O researchers with an individual mindset, given that personality traits are less valid than general cognitive ability in predicting individual job performance; however, these findings may illustrate the importance of the organization-level emergence of noncognitive and contextualized (vs. cognitively loaded) traits in enabling collective processes such as communication, cohesion, trust, coordination, and cooperation among members that directly influence collective (organizational) performance (see Ployhart, Reference Ployhart2021).
Frequency and Correlations With Firm Performance Outcomes
Note. DV = dependent variable; EBIT = earnings before interest and taxes; 04 = 2004; 04-07 = average of 2004 through 2007.
In the eyes of I-O researchers, one common limitation of Kim and Ployhart’s (Reference Kim and Ployhart2018) and Terpstra and Rozell’s (Reference Terpstra and Rozell1993) work is that the authors examined only the use/adoption (vs. nonuse/adoption) of the selection procedures mentioned above, not the degree of intensive use of each of the selection procedures. In fact, organizations that use the same selection procedure (can) use it in very different ways. For example, some organizations use a personality test as the first hurdle to screen out a large number of applicants, whereas others use it “as supplementary information about job applicants during the employment interviews by managers and/or top management team” (Oh etal., Reference Oh, Han, Holtz, Kim and Kim2018, p. 1353). Thus, the correlations reported above and in our Table 1 must have underestimated the performance benefit of each of the selection practices. In summary, the aforementioned findings, despite the methodological limitations mentioned above, highlight the peril of inferring organizational differences (organization-level findings) from individual differences (individual-level findings) in personnel selection.
Before closing, we would like to discuss a promising future research agenda on organizational differences in personnel selection. Recent SHRM research has shown that, “while the focus is on organization-level outcomes, the processes through which HR practices impact firm performance take place within individuals,” thus emphasizing the importance of human capital resources, an organizational-level emergence of individual differences such as knowledge, skills, abilities, and other attributes such as personality traits (Wright etal., Reference Wright, Nyberg, Ployhart, Buckley, Wheeler and Halbesleben2018, p. 155). Thus, HR practices and human capital sources should be distinguished, although prior SHRM research implicitly equated them (Ployhart & Hale, Reference Ployhart and Hale2014). As discussed above, the use of more selective selection system is more strongly correlated to organizational performance (Combs etal., Reference Combs, Liu, Hall and Ketchen2006; Kim & Ployhart, Reference Kim and Ployhart2018). However, what directly contributes to organizational performance is not the use of more selective selection system itself but what the use of such a selection system produces (i.e., higher quality human capital resources).Footnote 2
However, SHRM or multilevel I-O research has yet to integrate these two streams of research. For example, we need to examine the mediating role of human capital resources between HR practices and organizational performance. As Wright etal. (Reference Wright, Nyberg, Ployhart, Buckley, Wheeler and Halbesleben2018) noted, “a failure to do so (or even merely dabbling in one or the other) can lead to research that overly simplifies the causal explanation.” (p. 149). Against this backdrop, one interesting and important research agenda for organizational-level research on personnel selection, for example, is “Does more selective use of personality tests in personnel selection lead to a higher organizational-level mean and lower variance in job-relevant personality traits (i.e., personality-based human capital resources) and, subsequently, higher organizational performance (e.g., lower turnover rates and higher productivity and profitability)?” This type of research will help bridge micro I-O and macro SHRM research and enhance the status of personnel selection research by demonstrating its direct relevance to organizational performance and competitive advantage (Oh etal., Reference Oh, Kim and Van Iddekinge2015). We believe that multilevel I-O researchers are well suited to address this research question by taking advantage of their rigorous training in measurement and data analysis—as long as they are willing to adopt an organizational mindset and learn from SHRM research.
In a recent focal article published in this journal, Rogelberg etal. (Reference Rogelberg, King and Alonso2022) provided an insightful discussion about a long-overdue issue in I-O psychology that our research findings do not reach the public. In their view, what underlies this issue is that “our published articles do not make much attempt to truly connect and translate” to the extent that “managers effectively consume our research and translate it into practice.” Although this line of reasoning makes sense, another critical reason is that I-O research is often too micro in the eyes of organizational decision makers, thus limiting its generalizability and usefulness at the organizational level. Accordingly, as discussed in Schneider and Pulakos (Reference Schneider and Pulakos2022), one meaningful way “to make I-O research more effective in practice” (p. X) is “expanding our research focus from predominantly individual differences to studying organizational differences” (p. X) and paying “enhanced attention to the organization level of analysis as our frame of reference for research” (p. X). We cannot agree more with their proposal because it is also central to the identity of I-O psychology grounded on the scientist–practitioner model.
In conclusion, there is a significant shortage of organization-level research on personnel selection. In fact, SHRM research has replaced I-O research when HR managers (and consultants) discuss selection procedures/practices with organizational decision makers. Although this is an unfortunate missed opportunity, it is not lost forever. Echoing Schneider and Pulakos (Reference Schneider and Pulakos2022), we urge that I-O researchers, particularly those in personnel selection adopt an “organizational mindset” and conduct more research on organizational (vs. individual) differences in selection practices/procedures using the very methodological rigor that allows them to have a special status in studying individual differences in whatever that be.
