The scientist–practitioner model in industrial and organizational (I-O) psychology suggests a permeable boundary between science and practice. As Rupp and Beal (Reference Rupp and Beal2007) noted, “Practitioners should look to the scientific literature for guidance on setting up effective workplace systems, and scientists should take their cues from practitioners in identifying issues relevant to employee well-being and organizational effectiveness” (p. 36). Although this model was created several decades ago (Benjamin & Baker, Reference Benjamin and Baker2000), there remains a documented and persistent divide between science and practice (McHenry, Reference McHenry2007). For example, a review of 5,780 articles published in Journal of Applied Psychology (JAP) and Personnel Psychology (PPsych) from 1963 to 2007 ascertained that much I-O psychology research does not address current societal issues (Cascio & Aguinis, Reference Cascio and Aguinis2008). The divide does not seem to be narrowing (Colella, Hebl, & King, Reference Colella, Hebl and King2017), prompting Woodwark and MacMillan (Reference Woodwark and MacMillan2014) to call the issue of the “growing gulf between researchers and practitioners . . . exigent” (p. 324).
Directly related to the relationship between science and practice, the issue of scholarly impact has received substantial attention, but the majority of this work focuses on measuring impact only on the work of other researchers rather than on practitioners (Certo, Sirmon, & Brymer, Reference Certo, Sirmon and Brymer2010; Kozlowski, Reference Kozlowski2017; McNally, Reference McNally2010). Although the Society for Industrial and Organizational Psychology (SIOP) lists the advancement of “the science, practice, and teaching of industrial-organizational psychology” as its mission (SIOP, 2015), only science seems to be explicitly and consistently measured and rewarded for academics (Gomez-Mejia & Balkin, Reference Gomez-Mejia and Balkin1992). For many academics, scholarly publications and citations drive important rewards such as appointment, tenure, and promotion decisions. For I-O psychology programs, research impact—the number of publications in “A” journals and citations received by those articles in other academic journals—affects program rankings, reputation, monetary resources, and the future student applicant pool (Beiler, Zimmerman, Doerr, & Clark, Reference Beiler, Zimmerman, Doerr and Clark2014; Salter, Allen, Gabriel, Sowinski, & Naidoo, Reference Salter, Allen, Gabriel, Sowinski and Naidoo2016). Overall, it seems that pluralistic definitions of scholarly impact and the assessment of contributions to practice and teaching remain an afterthought (Aguinis, Shapiro, Antonacopoulou, & Cummings, Reference Aguinis, Shapiro, Antonacopoulou and Cummings2014).
Another aspect of the ongoing debate regarding the science–practice divide involves the education and training of future I-O psychology professionals—both practitioners and researchers. As an applied science, I-O psychology education seeks to influence both science and practice (Weathington, Bergman, & Bergman, Reference Weathington, Bergman and Bergman2014). Byrne et al. (Reference Byrne, Hayes, Mort McPhail, Hakel, Cortina and McHenry2014) recently pointed to I-O psychology students’ grounding in a “psychological base” as a source of competitive advantage (p. 8), and Aguinis, Bradley, and Brodersen (Reference Aguinis, Bradley and Brodersen2014) noted that although many I-O psychology researchers are moving to business schools, many have been trained using knowledge drawn from I-O psychology sources, a claim also echoed by Tett, Brummel, Simonet, and Rothstein (Reference Tett, Brummel, Simonet and Rothstein2014). So, it seems that the current zeitgeist is that future I-O psychology professionals are mostly trained in psychology. However, is it correct to make this assumption? What sources and authors constitute the initial knowledge base for I-O psychology professionals (both practitioners and researchers) of the future—those students enrolled in an introductory I-O psychology course?
Present Study and Research Questions
The goal of our study is to investigate what sources (e.g., scholarly journals, edited books, popular press publications), individual items (i.e., articles, book chapters, books), and authors are cited in some of the most widely used I-O psychology textbooks. Most future I-O psychology practitioners and researchers initially enroll in an introductory I-O psychology course during their junior or senior year of undergraduate studies, making introductory I-O psychology textbooks their first in-depth exposure to the field. Unlike journal articles, which primarily influence a smaller community of current and future researchers, I-O psychology textbooks influence the knowledge base of exponentially larger numbers of future practitioners and researchers.
Our results have implications regarding several issues that are currently debated vigorously: the science–practice divide (Cascio & Aguinis, Reference Cascio and Aguinis2008); how to define, measure, and reward scholarly impact (Aguinis, Suarez-González, Lannelongue, & Joo, Reference Aguinis, Suarez-González, Lannelongue and Joo2012); the movement of I-O psychology researchers to business schools (e.g., Aguinis, Bradley, et al., Reference Aguinis, Bradley and Brodersen2014); and the future of I-O psychology as a field (Aycan, Reference Aycan2014). For example, if textbooks refer to sources other than traditional peer-reviewed academic journals, this would offer evidence that the science–practice divide does not develop until later in the career of I-O psychologists. As a second illustration, if the authors cited most frequently in academic journals do not overlap with authors cited in textbooks, this would indicate that the knowledge base included in textbooks is not consistent with the most influential scholarly developments. As a third potential contribution of our study, if researchers housed in business schools produce more of the knowledge disseminated in introductory I-O psychology courses than their colleagues in I-O psychology programs, this would provide further evidence regarding the movement of I-O psychologists, and even I-O psychology, to business schools. In total, we addressed the following specific research questions about the relative influence of sources, individual articles and book chapters, and authors:
Research Questions About the Influence of Sources
-
Research Question 1 (RQ1): Which are the most frequently cited sources in popular I-O psychology textbooks?
-
Research Question 2 (RQ2): Among the most-cited sources, what is the proportion of academic publications compared to other types of sources?
-
Research Question 3 (RQ3): Among the most frequently cited academic sources, what is the proportion of academic I-O psychology sources compared to academic sources originating outside of I-O psychology?
Research Questions About the Influence of Individual Journal Articles and Book Chapters
-
Research Question 4 (RQ4): Which are the most-cited articles and book chapters in popular I-O psychology textbooks?
-
Research Question 5 (RQ5): Among the most-cited articles and chapters, what is the proportion of academic journal articles compared to other sources, and in what fields have they been published?
-
Research Question 6 (RQ6): Among the most-cited articles and chapters, how does the coverage of topics in I-O psychology textbooks compare to the coverage of the same topics in journals?
-
Research Question 7 (RQ7): What are the publication dates of the most-cited articles and chapters?
Research Questions About the Influence of Authors
-
Research Question 8 (RQ8): Who are the most frequently cited authors in popular I-O psychology textbooks?
-
Research Question 9 (RQ9): Among the most-cited authors, what proportion work in I-O psychology versus business school programs?
-
Research Question 10 (RQ10): Among the most-cited authors, what is the relation between their citations in textbooks and their citations in academic journals (i.e., impact on the academic literature)?
Method
Textbook Selection
We used three steps to identify the most popular and widely used I-O psychology textbooks. First, we searched the textbook section of Amazon.com, the world's largest online retailer (Li, Reference Li2015), and one of the largest retailers of textbooks (Mosendz, Reference Mosendz2014). We conducted individual searches using the subject area as keywords (e.g., “industrial organizational psychology textbook,” “organizational psychology textbook”). We excluded any results that were not specifically written to be used as introductory textbooks (e.g., Cascio & Aguinis, Reference Cascio and Aguinis2011) or that focused on narrower subfields (e.g., Lowman, Reference Lowman2006). As the number of editions published is an indicator of longevity, and therefore accumulated influence and popularity of a textbook, we only included books in at least their second edition. This process generated five books.
Second, we queried seven I-O psychology faculty at a large, private, mid-Atlantic university for the name of the textbook they used in their classes. We cross-referenced responses with the list we had compiled through Amazon.com and found all five textbooks mentioned by the faculty were already on our list.
Third, we examined the recommended I-O psychology textbook lists at a different large mid-Atlantic university (public), a large Midwestern university (private), and a large southwestern university (public). After examining the textbooks for the first two universities, we found that most of the recommended textbooks for these schools were already included on our list. However, our examination uncovered one additional textbook, which we added, taking our total to six textbooks. Upon examining the listings for the third university (southwestern), we reached saturation (Glaser & Strauss, Reference Glaser and Strauss1967) and found no additional recommended textbook not already included in our list. Table 1 lists the textbooks we reviewed and analyzed in our study.
List of Textbooks Reviewed and Analyzed in This Study
Note: I-O = Industrial-organizational.
Data Collection and Accuracy Checks
We used two different methods to collect our data. For three textbooks (Levy, Reference Levy2017; Muchinsky & Culbertson, Reference Muchinsky and Culbertson2016; and Riggio, Reference Riggio2013), we scanned the endnotes and references into PDF files using a high-resolution scanner. To make the data searchable, we conducted an optical character recognition (OCR) operation using Adobe Acrobat Pro software. For the other three textbooks (Aamodt, Reference Aamodt2016; Landy & Conte, Reference Landy and Conte2016; and Spector, Reference Spector2017), we obtained editable Word documents with the references directly from the authors. Next, we created a transcription template in Excel to capture the data from each references file and a detailed guide on how to transcribe different entries (e.g., book chapters versus journal articles). For each entry in endnotes or references, we extracted the following information: last name(s) of author(s); first name(s) of author(s); year of publication of entry; title of article/book chapter/publication (as applicable); journal/book/source. Multiple entries of the same article/book chapter in the same textbook (e.g., in endnotes and in references) were only counted once.
Next, we used six coders to create our database. The coders were the second, third, fourth, fifth, and sixth authors, and a freelancer whom we recruited from the Internet freelancing website Upwork.com (see Aguinis & Lawal, Reference Aguinis and Lawal2013, for a review of Internet freelancing). To select the Upwork coder, potential freelancers were provided with an abbreviated list of references from one of the textbooks and asked to submit a sample transcription. We reviewed this sample and clarified questions and errors with the freelancers. The freelancer who successfully completed the sample transcription was hired on an hourly basis. All coders received a copy of the transcription guide that provided examples of how to code different entries, the references for one of the textbooks, and an Excel file in which to enter the transcribed data. During transcription, we corrected obvious errors in the textbooks’ references sections (e.g., Kozlowski, S. W. J. listed as Kozlowski, S. J. W.).
Once the coders completed transcription of the textbook assigned to them, they created an Excel file with the transcribed data. The second author then conducted a preliminary check of the work by randomly inspecting the transcription of 10% of all entries. If a discrepancy was found in the submitted data file compared to the textbook's references list, the coder was asked to recheck the Excel file and correct discrepancies. The coders invested approximately 400 hours of work to transcribe the data from the PDF and Word files into Excel.
Following transcription, we conducted a second round of quality checks. For each textbook, each coder independently inspected all the transcribed entries for 20 randomly chosen authors against the textbook's references. In all, we inspected 903 entries during this process and found 18 errors, for an error rate of 2%. Most of these errors were due to the inability of the optical character recognition software to distinguish between letters (e.g., Yukl, G. A. scanned as Yuki, G. A.).
Next, with the data for each of the six textbooks quality checked, we concatenated the six Excel files into a single, master database. To ensure the integrity of the database, each coder independently concatenated the data from the individual files from each of the six textbooks into his or her separate master database. We then compared the results of our analysis of the most frequently cited authors, journals, and articles from the six separate master databases concatenated by each coder and found that they matched perfectly. As a final quality control step, we individually checked the entries for all articles and journals in the database, as well as the top-500 most-cited authors.
Although, as described above, we conducted extensive checks, given the size of our database, it is possible that some spelling errors may exist in some author names (i.e., last names and first and middle initials). However, given our accuracy-check procedures, these errors are random in nature, likely to be minimal, and therefore unlikely to change our substantive conclusions. Our final database of the endnotes and references for all six textbooks contains 8,372 rows of information, including individual items with multiple citations each. The database contains 6,654 unique published items (e.g., articles, book chapters), drawn from 1,682 unique sources (e.g., journals, books), and authored by 8,603 unique individuals with at least one citation each. Obviously, many of the items have multiple coauthors. So, cumulatively, these 8,603 unique authors are cited a total of 19,473 times when counting all the coauthors for each item in the database.
Results
Most-Cited Sources
To answer RQ1, we identified the top-100 most-cited sources, including those with equal numbers of citations. This selection procedure led to the 110 sources listed in Table 2. Each of these sources received at least seven citations; that is, on average they were cited more than once per textbook. As has been found repeatedly in the past, the distribution of citations is right heavy tailed, meaning that a relatively small number of sources accounts for a disproportionately large number of citations (e.g., Podsakoff, MacKenzie, Bacharach, & Podsakoff, Reference Podsakoff, MacKenzie, Bachrach and Podsakoff2005). Not surprisingly then, the top-110 sources listed in Table 2, which include less than 7% of the total number of sources (i.e., 1,682), accounted for 72% (i.e., 5,989) of the total number of citations (i.e., 8,372). So, the sources included in Table 2 are substantially more influential than the rest.
Top-110 (i.e., 6.5%) Most-Cited Sources in Popular Industrial-Organizational (I-O) Psychology Textbooks (Out of a Total of 1,682 Unique Sources)
Note: Sources are ranked by number of citations in I-O psychology textbooks. Sources with equal numbers of citations are listed alphabetically and assigned the same rank. Sources are classified as “Academic” if they are currently included in the Web of Science Journal Citations Report (JCR) database. JCR classifications are as of March 12, 2017. B = business and/or management only; APL = psychology-applied only; B/APL = business/management/applied psychology; B/APL/PO = business/management/applied psychology/other psychology; PO = psychology-other (psychology, social psychology, and multidisciplinary psychology); Other = nonpsychology or business related academic sources.
a Citation counts include both past and current names for these sources.
The two oldest and most established applied psychology journals (i.e., JAP and PPsych) are ranked #1 and #2, respectively. Table 2 also reveals the presence of practitioner publications such as HR Magazine (#20) and Harvard Business Review (#50), and popular press sources such as the New York Times (#34). In addition, Table 2 includes “bridge” journals, which typically feature articles authored by academics but target both academic and practitioner audiences (e.g., Human Resource Management is #35 and Organizational Dynamics is #47). In addition, this list of top-cited sources includes 25 edited volumes and one textbook.
Regarding RQ2, Table 2 distinguishes sources that are “academic journals.” We made this distinction based on whether a source is indexed by the Web of Science (WoS) database, which includes traditional peer-reviewed academic journals. We recognize that some publications may not be listed on the WoS database but nevertheless be peer-reviewed (to some extent) or academic in nature (e.g., publication targeting practitioners but also academics). But, in the interests of transparency and replicability of our procedures, we decided to use the WoS clear-cut inclusion criterion. Based on results in Table 2, 39% of the 110 most-cited sources are not academic journals. These sources account for 14% of the total number of citations accumulated by the 110 most-cited sources (i.e., 859 out of a total of 5,989). To answer RQ3, we used the WoS Journal Citations Reports (JCR) categories assigned to a journal. Journals were considered purely I-O psychology related (APL) if they were categorized only in the psychology-applied category by JCR (e.g., Applied Psychology: An International Review; Journal of Vocational Behavior), purely business-related (B) if they were categorized only in either or both of the business and management categories by JCR (e.g., Academy of Management Journal; Administrative Science Quarterly), and cross-disciplinary (B/APL and B/APL/PO) if they were categorized in both of the psychology-applied and business or management categories by JCR (e.g., JAP; Journal of Management; Journal of Organizational Behavior). Using these categories, other psychology journals account for the largest percentage (36%) of the most-cited academic sources listed in Table 2, followed by cross-disciplinary journals (29%), purely business journals (18%), purely I-O psychology journals (12%), and other nonpsychology or business related journals (6%). The top seven sources account for more than half of the total citations drawn from the most-cited sources in Table 2 (i.e., 3,061 out of a total of 5,989), providing further evidence of the right heavy tail of the distribution. These top seven comprise six cross-disciplinary journals and one purely business journal.
Most-Cited Articles and Book Chapters
Regarding RQ4, Table 3 Footnote 1 shows the most frequently cited articles and book chapters. The first six entries in Table 3 show items that have been cited in all six textbooks analyzed (i.e., total of six citations each). These items were published in a mix of cross-disciplinary journals (n = 3), edited volumes (n = 2), and an other psychology journal (n = 1). Of these six, three were published more than 40 years ago (i.e., Adams, Reference Adams and Berkowitz1965; Dansereau, Graen, & Haga, Reference Dansereau, Graen and Haga1975; French & Raven, Reference French, Raven and Cartwright1959). The other three include two meta-analyses (i.e., Judge, Thoresen, Bono, & Patton, Reference Judge, Thoresen, Bono and Patton2001; Van Iddekinge, Roth, Raymark, & Odle-Dusseau, Reference Van Iddekinge, Roth, Raymark and Odle-Dusseau2012) and a qualitative review (i.e., Baldwin & Ford, Reference Baldwin and Ford1988).
Top-59 Most-Cited Articles and Book Chapters in Popular Industrial-Organizational (I-O) Psychology Textbooks (Out of a Total of 6,654 Unique Articles and Book Chapters)
Note: Sources are ranked by number of citations in I-O psychology textbooks. Articles with equal numbers of citations are ranked alphabetically and assigned the same rank. Sources are classified as “academic journals” if they are currently included in the Web of Science Journal Citations Report (JCR) database. JCR data is as of March 12, 2017. JAP = Journal of Applied Psychology. TIP = The Industrial-Organizational Psychologist. Due to space considerations, only the top-59 articles (each with four or more citations) are listed. However, results reported in text are based on the top-219 articles (each with three or more citations). A full list of these articles is available from the authors upon request.
To answer RQ5, we identified all items with at least three citations each (i.e., cited by at least half the textbooks analyzed in our study). Journal articles account for 93% (i.e., 203 out of a total of 219) of total citations among the most-cited items. Using the same categories defined in the previous section, 77% (169) of the top-219 most-cited items were published in cross-disciplinary journals, 10% (21) in other psychology journals, 5% (12) in edited volumes, 3% (7) in purely business journals, and 2% (5) in purely I-O psychology journals.
To answer RQ6, we drew upon the work of Cascio and Aguinis (Reference Cascio and Aguinis2008), who conducted a content analysis of 5,780 articles published in JAP and PPsych from January 1963 through May 2007 and classified each article as addressing primarily one of 15 broad topical areas (e.g., job analysis, predictors of performance, motivation and work attitudes, leader influences). The second, third, and sixth author independently categorized all 219 top-cited items according to the 15 topical areas. We compared the categorizations using a simple matching function in Excel to determine the overlap between independent selections. In terms of intercoder agreement, results indicated that 85% of the items in each coder's independently categorized list was the same as those selected by the other coders. The correlation between the popularity of topics as addressed in textbooks and the journals examined by Cascio and Aguinis (Reference Cascio and Aguinis2008) is r(15) = .78, p = .001. The top three most popular topical areas in I-O psychology textbooks are (1) work motivation and attitudes (16%), (2) predictors of performance (14%), and (3) leader influences (11%). For I-O psychology journals, Cascio and Aguinis (Reference Cascio and Aguinis2008) reported that the top three domains are (1) methodology–psychometric issues (21%), (2) work motivation and attitudes (15%), and (3) performance measurement–work outcomes (14%). Thus, there is overlap between the broad content areas covered by textbooks and journal articles, although there is a slight difference in terms of the frequency of particular topics. For example, the issue of leader influences is almost twice as likely to be addressed in textbooks (11%) than in journal articles (6%).
Finally, for RQ7, we examined the publication dates for the 219 most-cited articles and book chapters. About 66% of these articles and book chapters have been published since 1997.
Most-Cited Authors
Regarding RQ8, Table 4 lists the most-cited authors. As mentioned earlier, the distribution of citations is right heavy tailed such that a relatively small number of authors accounts for a disproportionately large number of citations. Accordingly, Table 4 lists 178 authors with 14 or more citations, who constitute 2% of the total number of unique authors cited at least once (i.e., 8,603) but account for 22% of the total number of citations for all authors across entries in the I-O psychology textbooks analyzed (i.e., 4,268 citations out of a total of 19,473 cumulative author citations).
Top-178 (i.e., 2.0%) Most-Cited Authors in Popular Industrial-Organizational (I-O) Psychology Textbooks (Out of a Total of 8,603 Unique Authors With at Least One Citation Each)
Note: Authors are ranked in decreasing order of number of citations in I-O psychology textbooks, and then listed in decreasing order of number of Web of Science citations. Authors with the same number of textbook citations are assigned the same rank. WoS = Web of Science. I-O = industrial-organizational. HRM = human resource management. OB = organizational behavior. ILR = industrial and labor relations. Web of Science citations are as of March 18, 2017. N/A = not available, and is used to indicate authors for whom we were unable to obtain the relevant information.
a The numbers of textbook citations for authors of the textbooks analyzed are a sum of the total and mean number of citations in the other five textbooks analyzed. Including raw citation counts instead of the sum and mean number of citations in the other five textbooks would affect rankings such that Paul E. Spector's rank would change from 11 to 3 (from 48 to 63 citations); Paul E. Levy's rank would change from 65 to 16 (from 22 to 41 citations); and Frank J. Landy's rank would change from 94 to 17 (from 19 to 40 citations). Ronald E. Riggio and Michael G. Aamodt would change from not being included in this table to being ranked 65 (with 22 citations) and 117 (with 17 citations), respectively.
Our initial list of the top-178 most-cited individuals included some of the authors of the textbooks we analyzed. Accordingly, to take into account self-citations (i.e., textbook authors citing their own work), we used the same procedure as Diener, Oishi, and Park (Reference Diener, Oishi and Park2014). Specifically, we used the following formula to account for the impact of self-citations of textbook authors:
Total Citations = Total number of citations for textbook author in other five textbooks + Mean number of citations for textbook author in other five textbooks
For example, the number of total citations listed in Table 4 for Paul E. Spector is 48. This is the sum of the number of times his work was cited in the other five textbooks (n = 40) and the mean number of citations of his work in those textbooks (n = 8).
Regarding RQ9, most of the top-178 most-cited authors in Table 4 (58%) are affiliated with business schools, a little more than one-third (34%) are affiliated with I-O psychology programs, a small number (7%) are in industry, and a few (1%) are affiliated with an academic program other than business or I-O psychology. To answer RQ10, we collected citation information using WoS for each of the most-cited authors listed in Table 4. Note that WoS citations are based on citations in journals included in the WoS database, which excludes textbooks (Aguinis et al., Reference Aguinis, Suarez-González, Lannelongue and Joo2012). The correlation between textbook citations and WoS citations is r(178) = .37, p < .01.
Supplemental Analyses
An anonymous reviewer commented that it may be informative to also determine the academic training of the most-cited authors, which might be an additional indication of who influences the I-O psychology knowledge base included in textbooks. Accordingly, we collected information on the discipline in which the most-cited authors received their doctoral degree, and this information is also included in Table 4.Footnote 2 Results show that 54% (94) of the top-178 most-cited authors received their doctoral degree in I-O psychology, 20% (35) in business (organizational behavior, business, management, marketing, and human resources), 10% (18) in general psychology, and 9% (16) in social/organizational psychology. Other doctoral fields include industrial and labor relations (n = 3); clinical psychology and experimental psychology (n = 2 each); and economic and social statistics, personality psychology, physical education/biomechanics, and psychometrics (n = 1 each).
Also related to the most-cited authors, an anonymous reviewer asked “whether there is any evidence that the most-cited authors (who were trained as I-O psychologists but now reside in business schools) have sold-out . . . sold out in the sense of no longer remaining affiliated or actively involved in the field (e.g., through conference participation).” To address this issue, we collected information on SIOP and Academy of Management (AOM) membership for each of the nondeceased most-cited authors using the SIOP and AOM online membership directories (as of March 14, 2017). Because membership in SIOP or AOM only requires a financial commitment in the amount of the annual fees, we also collected data on the number of peer-reviewed presentations the most-cited authors authored or coauthored in the past decade (i.e., 2007–2016) using the online conference programs available on the SIOP and AOM meetings websites. Regarding membership, 134 authors are members of SIOP as compared to 108 who are currently members of AOM. Additionally, 36 are exclusively members of SIOP, whereas 10 are exclusively members of AOM.Footnote 3 Second, regarding presentations at the annual meetings, Figure 1 includes a graphical representation of the total number of peer-reviewed presentations per year in which the most-cited authors participated.
Total number of peer-reviewed presentations per year in which the most-cited authors in industrial-organizational psychology textbooks included in Table 4 participated at the annual meetings of the Society for Industrial and Organizational Psychology and the Academy of Management. Dashed lines represent linear trends.
Discussion
We discuss implications of our findings for issues currently debated in I-O psychology and related fields: the science–practice divide; how to define, measure, and reward scholarly impact; and the future of I-O psychology as a field, including considerations about the movement of I-O psychology researchers to business schools and the sustainability of I-O psychology programs in psychology departments.
Implications for the Science–Practice Divide
The scientist–practitioner model espoused by the field of I-O psychology implies a synergistic relationship between science and practice that benefits both parties. Previous research has examined the impact of scholarly work on subsequent scholarship, but we do not know what sources of knowledge, which specific articles, and which authors are most often referred to in I-O psychology textbooks, which is the first in-depth exposure to the field for most future I-O psychology professionals.
Compared to previous research, our results are encouraging regarding the scientist–practitioner model. For example, 39% of the 110 most frequently cited sources mentioned in I-O psychology textbooks are nonacademic journals. Moreover, these top 6.5% cited sources (i.e., 110 out of a total of 1,682) include practitioner publications such as HR Magazine (#20) and Harvard Business Review (#50), popular press sources such as the New York Times (#34), and bridge journals such as Human Resource Management (#35) and Organizational Dynamics (#47). These results show that future I-O psychology practitioners and researchers are exposed to both scholarly research (as published in academic journals) and the practical implications of such research (as published in practitioner publications and bridge journals). Thus, future I-O psychology professionals—those enrolled in introductory I-O psychology courses—are exposed to the practical side of organizational life, indicating that the divide may develop later, perhaps after graduates obtain employment as either practitioners or researchers.
Our analysis of the relative frequency of cited sources also revealed that although non-academic sources are commonly cited in I-O psychology textbooks, they account for a minority (i.e., 14%) of the total citations drawn from the most-cited sources. So, I-O psychology textbooks refer to practitioner and bridge sources, but they draw more heavily from scientific findings published in academic journals to shape the knowledge base upon which future I-O psychology professionals will rely. Another interesting finding is that 38% (84) of the top-219 most-cited articles are meta-analyses, which is consistent with results from Aguinis, Dalton, Bosco, Pierce, and Dalton (Reference Aguinis, Dalton, Bosco, Pierce and Dalton2011) that meta-analyses are cited, on average, more often than primary-level studies. Therefore, future practitioners are educated using not only knowledge published in rigorous academic journals, but also accumulated knowledge that has been synthesized quantitatively. These findings are also an encouraging result regarding the scientist–practitioner model because they suggest that textbook content includes evidence-based knowledge to guide future practitioners’ decisions and actions.
Finally, the presence of bridge journals such as The Industrial-Organizational Psychologist (TIP), Human Resource Management, and Organizational Dynamics is particularly interesting. Straddling the line between academic journals (e.g., JAP and PPsych) and practitioner publications (e.g., HR Magazine), these sources speak to and feature contributions from both academics and practitioners. Although these journals do not have a high JCR impact factor,Footnote 4 they nevertheless feature prominently in introductory I-O psychology textbooks. Thus, these journals represent an outlet that academics can use to bridge the science–practice divide and impact both current and future practitioners.
Implications for Defining and Measuring Scholarly Impact
Our results regarding the most frequently cited authors in I-O psychology textbooks have important implications about how the field defines, measures, and rewards scholarly impact (Aguinis et al., Reference Aguinis, Suarez-González, Lannelongue and Joo2012). Recall that we examined the extent to which authors cited by other researchers (as measured by WoS citations) are also cited in textbooks. Based on the data in Table 4, the correlation between textbook citations and WoS citations is r = .37. This may not seem like a large effect because “only” 14% of variance in WoS citations is explained by textbook citations (i.e. r 2 = .137). But, this correlation is more than twice the size of the median effect size of r = .16 calculated by Bosco, Aguinis, Singh, Field, and Pierce (Reference Bosco, Aguinis, Singh, Field and Pierce2015) based on 147,328 correlations reported in JAP and PPsych from 1980 to 2010. In other words, there is considerable overlap between the authors who are cited most frequently by other researchers and authors who are cited most frequently in I-O psychology textbooks. These results provide evidence that these authors influence both other researchers (i.e., those who are publishing in peer-reviewed journals) as well as future I-O psychology professionals (i.e., those who are enrolled in introductory I-O psychology courses). Also, given their influence on the scholarly community and textbooks, the authors listed in Table 4 seem to be boundary spanners—individuals able to move across and have influence on different types of knowledge communities.
Our findings also lead to implications for the design of faculty performance management systems. For example, consider the case of a university that is particularly interested in undergraduate learning and education, and having an impact on future practitioners. Should this university reward the extent to which a faculty member publishes in outlets referred to in textbooks, even if they are not “traditional” scholarly outlets? Should tenure and promotion systems at those universities expand their journal lists to include bridge journals such as TIP, Human Resource Management, and Organizational Dynamics, which do not enjoy particularly high JCR impact factor scores (or do not have an impact factor score at all) but are nevertheless cited frequently in textbooks (ranked #14, #35, and #47, respectively)? As another particularly relevant example, consider the case of Industrial and Organizational Psychology: Perspectives on Science and Practice (IOP). Established in 2008, IOP’s 2015 JCR impact factor of 0.38 is admittedly smaller than that of other top-cited journals included in Table 2, such as Journal of Personality and Social Psychology (JPSP; impact factor = 4.74), Administrative Science Quarterly (ASQ; impact factor = 5.32), and Psychological Science (PS; impact factor = 5.48). However, IOP is cited just as often as JPSP in I-O psychology textbooks (both ranked #27) and more often than ASQ (ranked #31) and PS (ranked #56). Given these results, should universities with a mission to influence the undergraduate learning and education of future practitioners revise their reward systems to encourage more “star performers” in terms of their influence on textbooks, as suggested more generally by articles published recently (Aguinis & O'Boyle, Reference Aguinis and O'Boyle2014; Aguinis, O'Boyle, Gonzalez-Mulé, & Joo, Reference Aguinis, O'Boyle, Gonzalez-Mulé and Joo2016; Joo, Aguinis, & Bradley, Reference Joo, Aguinis and Bradley2017)? Our results provide a reminder of the need for the field to address such thorny questions in the years to come and the need for future research to produce knowledge on how to define and measure scholarly impact more pluralistically (Aguinis, Shapiro, et al., Reference Aguinis, Shapiro, Antonacopoulou and Cummings2014).
As an additional contribution of our study, we make our entire database available upon request. As we describe in more detail later in our article, making our database available will allow interested readers to conduct additional impact-based analyses that may be of particular interest and usefulness for various purposes. For example, university administrators will be able to use our database to search for individual faculty members in a particular university or department and learn about their relative impact in terms of the knowledge included in textbooks. Also, this same author-based search can be conducted for particular individuals across universities, such as a cohort of researchers who received their doctorates in the same year. The resulting information can be used for developmental as well as administrative purposes, such as the allocation of rewards and as additional information for other decisions such as tenure and promotion—particularly for those universities for which impact based on the information disseminated through textbooks is an important strategic objective. Similarly, journal editors will be able to use the database to compare the relative attention received in textbooks by their journal compared to others. This type of information can be useful as another indicator of impact—in concert with the more traditionally and typically used impact factor score, which focuses exclusively on citations in other academic journals.
Implications for the Future of I-O Psychology
Our results provide additional evidence regarding the movement of I-O psychologists to business schools. Specifically, Aguinis, Bradley, et al. (Reference Aguinis, Bradley and Brodersen2014) argued that the growing trend of I-O psychologists taking positions in business schools, as opposed to psychology departments, heralded important changes for the field. Results of our study indicate that some of the predictions made by Aguinis, Bradley, et al. (Reference Aguinis, Bradley and Brodersen2014) may already be coming true.
Our results indicate that 58% of the 178 most-cited authors in I-O psychology textbooks are affiliated with business schools, whereas only 34% are affiliated with psychology departments. This is a noteworthy result for two reasons. First, as shown in Table 1, six of the seven authors of the textbooks included in our study are or were (in the case of deceased authors) affiliated with a psychology department. Nevertheless, these psychology-department–affiliated textbook writers cited a majority of authors housed in business schools. Second, this trend holds even among the most-cited authors who received their PhDs in I-O psychology, with 47% working in business schools and 43% in psychology departments. This means future I-O psychology practitioners and researchers are exposed to more business-school–affiliated scholars via their textbooks. A potential implication of these results is that undergraduate students interested in pursuing doctoral studies may be drawn to the work authored by researchers in business schools found in their I-O psychology textbooks and then decide to seek admittance into a doctoral program in a business school (e.g., organizational behavior, human resource management) rather than an I-O psychology program. Clearly, if a large number of potential I-O psychology graduate students decide to enroll in business schools instead, this would be detrimental for the future pipeline of I-O psychologists with a doctoral degree and would affect the sustainability of I-O psychology programs within psychology departments.
Another challenge posed by our results is with regard to the knowledge base of I-O psychology itself. Regarding the movement of I-O psychologists to business schools, Allen, Eby, Weiss, and French (Reference Allen, Eby, Weiss and French2014) noted that “the real issue of concern is not brain drain but the impact that the immigration of the managerial sciences is having on the research published in I–O psychology journals” (p. 307). As our results show, although other psychology journals are cited somewhat more frequently than cross-disciplinary journals (36% vs. 29%) among the 110 most-cited sources listed in Table 2, cross-disciplinary journals account for two-thirds (66%; i.e., 3,381 out of a total of 5,130) of the citations from among these academic sources, whereas other psychology and purely I-O psychology sources are cited far less frequently (11%; 588 and 10%; 519, respectively). Moreover, 77% of the 219 most-cited articles were published in cross-disciplinary journals, compared to 10% for other psychology and 2% for purely I-O psychology journals. Thus, our results point to challenging questions regarding the future of I-O psychology and its relationship with other psychology specialty areas such as social psychology, as well as the field of psychology in general. Based on the knowledge summarized in I-O psychology textbooks, I-O psychology is much closer to business and management than social psychology and psychology in general.
Directly related to the aforementioned results and the movement of I-O psychologists and I-O psychology to business schools, based on information in Table 4, the mean graduation year of the top-178 most influential authors is 1982 and the median is 1983. So, overall, the most-cited authors received their doctorates about 34 years ago. As noted by an anonymous reviewer, it is likely that their doctoral training has influenced their thinking and research. But, we believe that a work context involving ongoing and regular interactions with colleagues and doctoral students mostly in business schools over a period of about 3.5 decades is likely a more powerful influence on their scholarship than the doctoral training they received about 34 years ago. Furthermore, if we consider the 25 authors in Table 4 who have received a doctorate since 1997 as midcareer scholars—given an average academic career length of about 40 years (Aguinis et al., Reference Aguinis, Suarez-González, Lannelongue and Joo2012)—we find that 68% (17) have earned an I-O psychology degree, 28% (7) have earned a business/management degree, and one author a general psychology degree. We draw two implications from these results. First, business-trained scholars comprise 20% (35) of the 178 most-cited authors in Table 4, but 28% (7) of the most-cited authors who are in the middle of their academic careers. This indicates that scholars trained in business are becoming increasingly more influential in shaping the knowledge base of I-O psychology. Second, whereas 20% (34) of the 178 most-cited authors in Table 4 were trained in social/organizational and general psychology, only 4% (1) of the midcareer scholars received such training. This provides further evidence that the knowledge base of I-O psychology is moving closer to business and management, and further away from social psychology and psychology in general.
On a perhaps more encouraging note for the sustainability of the field of I-O psychology in psychology departments, we found that despite moving to business schools, many of the most-cited authors continue to stay engaged with the I-O psychology community. For example, a slightly greater percentage of the most-cited authors are members of SIOP than of AOM. In addition, results in Figure 1 show that the most-cited authors participated in slightly more presentations at SIOP than AOM. However, the gap has almost closed in 2016, and the linear trend lines suggest that the number of AOM presentations by the top-cited authors may surpass those of SIOP in the near future.
Our results expand upon the work by Aguinis, Bradley, et al. (Reference Aguinis, Bradley and Brodersen2014), Byrne et al. (Reference Byrne, Hayes, Mort McPhail, Hakel, Cortina and McHenry2014), and Tett et al. (Reference Tett, Brummel, Simonet and Rothstein2014), who pointed to a psychology-driven knowledge base as an important strength for the field of I-O psychology. Our results indicate that I-O psychology students’ initial base of knowledge is much more likely to be influenced by cross-disciplinary academic journals than by research published in purely I-O psychology journals or other psychology journals. These results speak partially to the issue of the crisis of identity that is affecting I-O psychology today (Lefkowitz, Reference Lefkowitz2010; Ryan, Reference Ryan2003; Woodwark & MacMillan, Reference Woodwark and MacMillan2014). Students whose introduction to I-O psychology is shaped by cross-disciplinary sources and articles written by business school professors may view themselves more as “organizational researchers” than as “I-O psychologists.” This means that they would be equally, if not more, likely to draw upon knowledge from business schools for their own practice and research, aim to publish in business-related outlets, and consider career positions in business schools. As such, we conclude that the concern noted by Allen et al. (Reference Allen, Eby, Weiss and French2014) extends beyond research to questions about the future of I-O psychology as an independent field housed in psychology departments.
In addition to the origins of the knowledge base for I-O psychology, our comparison of the broad topical areas and authors referenced by the most-cited articles and book chapters with the areas identified by Cascio and Aguinis (Reference Cascio and Aguinis2008) in their review of research published in two premier I-O psychology journals reveals similarities between the two. This is not entirely surprising given that, after all, textbooks should rely on state-of-the-science knowledge as published in academic journals. Although there were some differences—for example the issue of leader influences is almost twice as likely to be addressed in textbooks (11%) than in journal articles (6%)—there is a high degree of overlap (a correlation of .78) between the broad content areas of textbooks and journal articles.
As an additional result, our study uncovered a troubling finding: the severe underrepresentation of female authors. For many years, the number of women earning a degree in I-O psychology in the U.S. has exceeded the number of men (National Center for Education Statistics, 2017). For example, in 2016, women earned 70% of all bachelor's degrees, 68% of all master's degrees, and 62% of all doctorates in the field of I-O psychology (National Center for Education Statistics, 2017). Despite this trend, our results show that women constitute only 17% (i.e., 29 out of a total of 178) of the most-cited authors listed in Table 4. In part, this could be due to the fact that, as mentioned earlier, the mean graduation year of the top-178 most influential authors is 1982, and the number of women in the field was much smaller then. Nevertheless, this is a troubling result that certainly deserves urgent consideration. In addition, results showed that only 15% (i.e., 25 out of a total of 178) of the most-cited authors listed in Table 4 in I-O psychology textbooks reside outside the U.S. This result is similar to findings by Cascio and Aguinis (Reference Cascio and Aguinis2008) regarding authors of JAP and PPsych articles, and, therefore, not entirely surprising. However, the growing internationalization of I-O psychology (Griffith & Wang, Reference Griffith and Wang2010) suggests that this is another area of opportunity for the field. Expanding the horizons of I-O psychology to include more international perspectives when training future practitioners and researchers can expand I-O psychology's contribution and practical impact by helping us better “inform the public about the purpose and importance of our field” (Rupp & Beal, Reference Rupp and Beal2007, p. 38).
Last, we also examined the dates of publication for the most-cited articles or book chapters and found that 66% of the top-219 most-cited articles and book chapters have been published since 1997. On one hand, the use of recent sources is encouraging as it implies that I-O psychology students are receiving the most updated knowledge. Thus, I-O psychology textbooks are fulfilling their role of educating future I-O psychology professionals with current research findings. However, personal experience as textbook writers and conversations with other textbook authors suggest that this drive for newness may also be motivated by the demands of textbook publishers, who often encourage the addition of “new” (and the deletion of “old”) references for successive editions of a textbook. As introductory I-O psychology classes are the gateway for future I-O psychology researchers, forsaking classics of I-O psychology in favor of recent articles, mostly authored by business school faculty and published in cross-disciplinary journals, textbooks may, unwittingly, be contributing to a lack of I-O psychology identity among I-O psychology graduates (Allen et al., Reference Allen, Eby, Weiss and French2014; Thoroughgood, Jacobs, & Caligiuri, Reference Thoroughgood, Jacobs and Caligiuri2014).
Limitations and Additional Future Directions
Although we analyzed popular textbooks that have broad impact, we readily acknowledge that there are other textbooks available, many of which address I-O psychology subdomains such as motivation, leadership, and ethics, among others. An analysis of those more specialized textbooks might produce different results from those obtained in this study. Thus, future research could examine the relative impact of sources and authors in more specialized domains.
Another potential limitation of our study is the use of citations as a measure of impact. Clearly, measuring impact and influence based on citations is an established methodological approach that has been and continues to be used for bibliometric research in many fields. For example, past research has used citation counts to assess the relative impact of researchers over time (e.g., Podsakoff, MacKenzie, Podsakoff, & Bachrach, Reference Podsakoff, MacKenzie, Podsakoff and Bachrach2008), the influence of journals (e.g., Podsakoff et al., Reference Podsakoff, MacKenzie, Bachrach and Podsakoff2005), and trends in the influence of different streams of research within a field (e.g., Ramos-Rodríguez & Ruíz-Navarro, Reference Ramos-Rodríguez and Ruíz-Navarro2004). However, a citation count is an imperfect measure of impact because it does not assess the reason why a particular source has been cited (Kacmar & Whitfield, Reference Kacmar and Whitfield2000; Zupic & Čater, Reference Zupic and Čater2015). For example, consider our own manuscript as an illustration of this point. We cited Li (Reference Li2015) and Mosendz (Reference Mosendz2014) to reference the primacy of Amazon.com as an online retailer of textbooks. Although these citations clearly influenced our work, they are not as influential in our conceptualization, research design, measures, and analytic procedures as the work by Aguinis et al. (Reference Aguinis, Suarez-González, Lannelongue and Joo2012) and Cascio and Aguinis (Reference Cascio and Aguinis2008). If we conduct a citation count of sources included in our manuscript, these four sources would receive one citation each, although they were not equally influential on our work. Additionally, the number of citations garnered by a source may also be affected by the longevity of the journal (i.e., how many years it has been published) and the number of issues it publishes per year. For example, in March 2017, JAP celebrated its centennial as a journal. In contrast, IOP has only been published since 2008. Accordingly, there is a much larger pool of JAP articles to be cited in textbooks compared to IOP and many other journals. However, this difference in publication history and frequency is not a methodological artifact but likely a substantive reason why certain journals are more impactful than others and should therefore not be corrected statistically. In short, although not always ideal, the use of citations offers an important initial assessment of relative impact.
Regarding future research, as an additional contribution of our study, we make our entire database available upon request. In addition to comparisons based on the citations of individuals and groups of researchers, our database can also be used to conduct additional analyses aimed at understanding why certain sources, articles, and authors are cited more than others. For example, researchers can use computer aided text analysis (CATA) to understand if particular theories and streams of research are more likely to be cited than others (McKenny, Aguinis, Short, & Anglin, in press). As another example of future use of the database, researchers can conduct a content analysis of the most-cited articles to determine if there are specific research designs, methodologies, or data analytical techniques that are more influential than others. Finally, future research could study the current and past affiliations, training, and socialization of the most-cited authors to understand if differences in these factors contributed to certain authors being more highly cited than others.
Our database is formatted to allow users to locate and synthesize data to answer these and other questions. Each row in our database refers to an entry from one of the six introductory I-O psychology textbooks analyzed. The first column lists the last name(s) of the author(s) of the textbooks. The next 12 columns list, in order of authorship, the names of the authors of the reference cited. These are followed by columns listing, in order, the year of publication of the reference, the name of the article/book chapter (as applicable), and finally the source of the reference. We offer the database formatted as a comma separated values (.csv) file for use with Excel or to import into a variety of statistical analysis software (e.g., R, SAS, SPSS). For example, in Excel, users can use the “Find All” function (accessed via the Ctrl+F keys) to quickly locate all entries for a particular author or journal. In sum, our database can be used to contribute to the discussion and understanding of the key features of impactful authorship and influence conducted by other researchers in the past (e.g., Hadani, Coombes, Das, & Jalajas, Reference Hadani, Coombes, Das and Jalajas2012; Judge, Cable, Colbert, & Rynes, Reference Judge, Cable, Colbert and Rynes2007; Podsakoff et al., Reference Podsakoff, MacKenzie, Bachrach and Podsakoff2005).
Conclusions
Our study went beyond the traditional examination of impact that has focused exclusively on citations in journal articles. Adopting Aguinis, Shapiro, et al.’s (2014) pluralist perspective allowed us to examine the extent to which different types of sources and authors influence the knowledge included in the most widely used I-O psychology textbooks—the initial and first in-depth knowledge base used to train future generations of I-O psychology practitioners and researchers. Our results expand upon and also offer useful information regarding the science–practice divide, how to define and measure scholarly impact, and the future of I-O psychology as a field.
Our results are encouraging regarding the transmission of the scientist–practitioner model to students who receive their first in-depth exposure to I-O psychology. Specifically, although the majority of citations in textbooks refer to academic journal articles, there is a prominent presence of nonacademic sources as well as bridge journals. These results clearly do not reflect the wide science–practice gap documented by past research focused on academic publications.
Results are also encouraging regarding the continued affiliation of the most-cited authors with I-O psychology because we found that such academics continue to remain members of SIOP and participate in SIOP conferences. So, although the majority of authors cited in I-O psychology textbooks are currently affiliated with business schools (in spite of their I-O psychology doctoral training), and they choose to publish in cross-disciplinary journals, they have not abandoned the field of I-O psychology.
The overlap between authors whose work is highly cited by other researchers and those whose work is most cited in textbooks is also encouraging. This shows that high-quality and rigorous scholarship, ostensibly the reason those articles are referred to by other researchers, also receives substantial attention in textbooks.
Our article offers an actionable system to measure scholarly impact more pluralistically—reaching beyond the traditional impact measure focused on citations by other researchers exclusively. In addition to results included in our article, we make the database available so that readers can conduct additional analyses and comparisons (e.g., between individuals, between departments, between schools, between universities). Finally, our results also lead to challenges regarding the future of I-O psychology in terms of its relationship with the broader field of psychology versus business and management, and the content of the knowledge disseminated in I-O psychology textbooks. Taken together, we hope our results will serve as a catalyst for the ongoing debates regarding the science–practice gap, the definition and assessment of scholarly impact, and the future of I-O psychology, including the movement of I-O researchers to business schools and the sustainability of I-O psychology programs in psychology departments.
