The focal article by Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) offers a rich brew of data and explication regarding the devilishly complicated concept of impact of sources (both science-based and practice-based), of industrial and organizational (I-O) psychology articles, and of authors of articles in the discipline of I-O psychology. The criteria developed to assess the impact of these are simple frequency counts indexing the number of times items within each domain are cited by leading introductory I-O psychology textbooks, which in turn leads to ranked lists. These are then used as a basis for answering numerous highly significant questions about the discipline of I-O psychology, including its scholarly and practice-based underpinnings, and the future prospects of the field.
Aside from the fun factor associated with ranked lists, it is imperative that these measures be evaluated on their merits. (Nevertheless in the spirit of fun this author is considering legally changing his name to “Number 32.”) This is particularly significant for the potential use of author citation data regarding impact of individual faculty members that would affect appraisals leading to outcomes such as tenure, promotion, and increases in compensation. As such this commentary will focus primarily on the last criterion, frequencies of author citations in I-O textbooks as a criterion of scholarly impact and will evaluate it as viewed through the lens of two classical articles, Wallace's (Reference Wallace1965) “Criteria for What” and Weitz's (Reference Weitz1961) “Criteria for Criteria.”
Before proceeding with that, I offer a few comments regarding other aspects of the focal article.
Issues Regarding Impact and the Future of I-O Psychology
A positive aspect of the focal article is the clear implication that impact is a multilayered concept, and as a result, multiple criteria are called for. Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) propose as one element of the requisite battery frequency of citations in introductory I-O psychology textbooks, those often used in introductory undergraduate I-O psychology classes, and have taken great care to ensure a database of high quality. The argumentation in support of this choice rests on the notion that these texts “constitute the initial knowledge base for I-O psychology professionals . . .” (Aguinis et al., Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 509) and “introductory I-O psychology textbooks [are] their first in-depth exposure to the field” (p. 509). The case for the criterion seems superficial; preferably the authors could have constructed a more thorough conceptual framework of impact as applied to sources, articles, and authors of articles, and then specified the place of frequency of citations in introductory I-O psychology texts in the structure. Moreover, it was not shown empirically or otherwise that introductory I-O psychology texts supply anything more to an undergrad's knowledge base than some notion at best that the field could be interesting or at worst that it fulfilled a degree requirement and would be forgotten after the course final. Nor was it shown that the information in the course textbook is remembered after entry into graduate school or a career.
As for the migration of I-O psychology to business schools, I think the connection between citation frequency of researchers affiliated with business schools, and frequency of citations of business-oriented sources on the one hand, and on the other the conclusion that “potential I-O psychology graduate students [will] decide to enroll in business schools instead” (Aguinis et al., Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 548) is an inferential leap of major proportions and lends itself to alternate interpretations. Business schools provide a core of knowledge that focuses on strategy at the top of the totem pole (though I for one wonder why when those in strategy have apparently not solved the riddle of what preordains business success or failure), finance, accounting, marketing, management, and human resources (much lower on the totem pole), including labor relations, compensation, and deployment. Doctoral-level training prepares graduates virtually exclusively for academic positions.
On the other hand, I-O psychology rests on a core of human development and cognition, motivation and emotion, neuroscience, psychological aspects of leadership, social and societal influences, differential psychology (i.e., study of individual differences, which masquerades these days as talent management), human communication, and group or team dynamics. As for methodology, I-O psychology graduates bring sophistication in such areas as laboratory research and psychometrics. All these sources of knowledge are clearly of benefit to business schools as Kahneman and Tversky have shown so convincingly (e.g., Lewis, Reference Lewis2017). Doctoral graduates are prepared for and may enter a diversity of fields, not just academics, including private and governmental organizations, consulting firms, research organizations, NGOs, and even their own businesses. Thus business schools should have a keen interest in keeping the pipeline from I-O psychology programs open. Students will continue to seek the kind of graduate training that suits their interests and career plans regardless of where a subset of I-O psychology graduates are affiliated. Indeed the fact that business schools are willing to pay premium rates to bring I-O psychologists on board is an added inducement to enroll in an I-O psychology program, not to mention the wider expanse of options available to them when they graduate.
One way to evaluate the sustainability of I-O psychology programs, which Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) suggest is threatened by the migration of I-O psychology to business schools, is to note whether substantial changes in numbers and quality of applicants to I-O programs have occurred. The program I am most familiar with at the University of South Florida (USF) has attracted the following numbers of applicants over the past 5 years, starting in 2013, for four to six entrants each year: 121, 175, 151, 136, 108. These data suggest no lack of substantial competitiveness in the program across all years, and those entering the program are consistently highly talented. Dr. Tammy Allen, the I-O psychology program director, reports that the overall quality of the applicant pool remains high. The declining numbers over the past 3 years, however, might be the beginning of a trend indicative of declining interest. Complicating any firm conclusions for USF's program are other factors such as changes in the deadline date for applying, which has become much earlier than in the past.
Dr. Clint Bowers, chairperson of the Psychology Department at the University of Central Florida, has graciously provided data on both its doctoral and terminal master's program applicant pools. Over 5 years beginning in 2013, the data are as follows with master's program applicant numbers in parentheses and respective overall GPAs following each applicant total: 100, GPA 3.87 (119, GPA 3.56); 102, GPA 3.6 (109, GPA 3.74); 108, GPA 3.66 (135, GPA 3.59); 90, GPA 3.65 (159, GPA 3.8); 108, GPA 3.79 (128, GPA 3.77). Entrants ranged from four to seven for the doctoral program and 13 to 15 for the master's program. No consistent downward trend in the number or quality of the applicant pool is evident.
By contrast with these two large public universities the Florida Institute of Technology (FIT) is a much smaller private institution. It also offers a terminal master's program in addition to the doctoral program as UCF does. FIT's number of applicants, courtesy of Dr. Lisa Steelman, the program director, over 5 years starting in 2013 are as follows, with master's program applicant numbers in parentheses: 57 (76), 58 (58), 73 (92), 55 (80), 54 (66) with 5 to 10 doctoral program entrants and 5 to 11 master's program entrants each year. Dr. Steelman reports that GREs and GPAs have been relatively stable over this period. Again as with USF, the recent decline in applicant numbers creates questions.
Overall these data, albeit quite preliminary, do not support any imminent existential threat to the sustainability of these I-O psychology programs, but they do suggest that the recent declines at USF and FIT need to be tracked in future. Perhaps the improvement of the world economy over the past few years with a concomitant increase in job opportunities is a more potent contributor to any observed decline in applicant numbers than the migration of I-O psychology to business schools. Of course a larger and more representative sample of programs nationally and even internationally would be a far better gauge of any threat to I-O psychology programs’ viability.
I do not doubt, however, that staffing I-O psychology programs while maintaining internal pay equity within a psychology department will continue to be a serious challenge. Perhaps, ironically, business schools, given the benefits of I-O psychology programs to business schools, will find a way to provide support to I-O psychology programs in some way as a means of responding to this challenge. For example they might provide part time appointments to I-O psychology faculty housed in psychology departments to supplement the lower salaries paid in those departments.
Frequency of Author Citations in Textbooks Through the Lenses of Wallace's (Reference Wallace1965) “Criteria for What” and Weitz's (Reference Weitz1961) “Criteria for Criteria”
Most pertinent to the issue at hand, namely the use of frequency of author citations in I-O texts, Wallace (Reference Wallace1965), whose argumentation is based within the psychometric tradition of the I-O field, railed against choosing criteria for correlational, particularly predictive, studies on the basis of expediency and practicality without consideration of a deeper conceptual basis. He answered his question by asserting that choosing criteria to facilitate understanding was the key if I-O was to assume its place as a true scientific discipline. Construct validity was seen as crucial to achieve this objective.
This search for understanding suggests that Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) have done a great service by advancing the question of the meaning of impact. However, it is clear that much more remains to be done to elucidate an understanding of the construct of impact and how authors’ citation counts in I-O psychology introductory texts factor in.
From a practical perspective, this lack of understanding also precludes successful performance management in the domain of academia. If we follow good practice in goal setting, we would specify for faculty quantity, quality, process requirements, deadlines, and/or cost considerations. This contrasts with my usual pronouncements to new faculty, for example when I served as chairperson of my department, namely, publish some articles in rigorous journals (not sure how many, not sure if textbook writing is valued); bring in some grant money; do not get slammed by student course ratings; make friends; and help out around the department, but not too much. This should lead to tenure and promotion. If I were to suggest that they pursue getting frequent citations in introductory I-O psychology textbooks, there is no clear path to achieve this. Certainly as Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) show, there is a moderate correlation between Web of Science citations and textbook citations, and authoring meta-analyses gives one about a 17% chance of cracking the most cited articles (84 articles among the 219 most cited were meta-analyses). But the path to success on this criterion is far from clear. (I suspect that if this becomes a performance requirement, I-O psychology textbook authors can count on tributes of various kinds, flowers and candy among them.)
Weitz (Reference Weitz1961) writes out of I-O psychology's experimental laboratory tradition, which was a bedrock in the emergence of I-O psychology as a science owing to the work of such pioneers as Hugo Munsterberg. Weitz identified three criteria for criteria: time, type, and level.
Time refers to how long after a manipulation of independent variables do we take our criterion measure. Type is the construct to be captured by the criterion. Level is what standard of performance must be attained to reach a desired outcome, such as mastery in a training context.
In the case of citation frequency we have substantial problems. Time after completion of graduate studies to begin counting citations would correspond to Weitz's time criterion. The frequency count used by Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) does not equate time in career across authors, nor does it sample across time periods for the count. The former metric would perhaps bias counts from textbooks in favor of older authors who have had more time to publish, whereas the latter would favor younger authors when textbooks are under pressure to cite new material. Using texts from different eras might for example place on the list luminaries such as Cronbach, Meehl, and Herzberg (who served as a very popular starting point for critiques and research in I-O psychology in his day), and might elevate luminaries like Guion to a much higher place on the list.
As for type, once again the issue of construct validity of the measure rises. Based on correlational data, there is some indication of overlap with research citations, but the data and theory on construct validity are sparse or lacking. We need to drill much deeper into the meaning of impact and likely explore multimethod ways to get at indices. For example, I have no doubt that Jacob Cohen was and is highly impactful by virtue of his work on multiple regression and analysis of variance (ANOVA) as two sides of the same coin, statistical power, frailties of null hypothesis testing, and kappa, but he does not appear on the list of the top 178. This is because his work is foundational for the research of others, but his articles do not get cited in I-O psychology textbooks.
Level poses great difficulty also in that merely having more citations than others does not inform about how much impact has been achieved; rather it merely indicates that one has more or less than others. Returning to the hypothetical example of counseling young faculty, a chair might say to the new faculty member that he/she needs to get more than similarly situated others, but how many are needed to get desired outcomes like tenure and promotion, or what constitutes excellence, cannot be specified.
Overall in my view Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) deserve much credit for pushing higher on the research agenda the question of impact, which incidentally deserves to be evaluated for I-O psychology practitioners as well as academics. Furthermore, they have provided a rich and carefully constructed data set to help advance the study of impact. Hopefully they and others will continue to pursue the resolution to this form of the eternal criterion problem.
You have
Access
The focal article by Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) offers a rich brew of data and explication regarding the devilishly complicated concept of impact of sources (both science-based and practice-based), of industrial and organizational (I-O) psychology articles, and of authors of articles in the discipline of I-O psychology. The criteria developed to assess the impact of these are simple frequency counts indexing the number of times items within each domain are cited by leading introductory I-O psychology textbooks, which in turn leads to ranked lists. These are then used as a basis for answering numerous highly significant questions about the discipline of I-O psychology, including its scholarly and practice-based underpinnings, and the future prospects of the field.
Aside from the fun factor associated with ranked lists, it is imperative that these measures be evaluated on their merits. (Nevertheless in the spirit of fun this author is considering legally changing his name to “Number 32.”) This is particularly significant for the potential use of author citation data regarding impact of individual faculty members that would affect appraisals leading to outcomes such as tenure, promotion, and increases in compensation. As such this commentary will focus primarily on the last criterion, frequencies of author citations in I-O textbooks as a criterion of scholarly impact and will evaluate it as viewed through the lens of two classical articles, Wallace's (Reference Wallace1965) “Criteria for What” and Weitz's (Reference Weitz1961) “Criteria for Criteria.”
Before proceeding with that, I offer a few comments regarding other aspects of the focal article.
Issues Regarding Impact and the Future of I-O Psychology
A positive aspect of the focal article is the clear implication that impact is a multilayered concept, and as a result, multiple criteria are called for. Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) propose as one element of the requisite battery frequency of citations in introductory I-O psychology textbooks, those often used in introductory undergraduate I-O psychology classes, and have taken great care to ensure a database of high quality. The argumentation in support of this choice rests on the notion that these texts “constitute the initial knowledge base for I-O psychology professionals . . .” (Aguinis et al., Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 509) and “introductory I-O psychology textbooks [are] their first in-depth exposure to the field” (p. 509). The case for the criterion seems superficial; preferably the authors could have constructed a more thorough conceptual framework of impact as applied to sources, articles, and authors of articles, and then specified the place of frequency of citations in introductory I-O psychology texts in the structure. Moreover, it was not shown empirically or otherwise that introductory I-O psychology texts supply anything more to an undergrad's knowledge base than some notion at best that the field could be interesting or at worst that it fulfilled a degree requirement and would be forgotten after the course final. Nor was it shown that the information in the course textbook is remembered after entry into graduate school or a career.
As for the migration of I-O psychology to business schools, I think the connection between citation frequency of researchers affiliated with business schools, and frequency of citations of business-oriented sources on the one hand, and on the other the conclusion that “potential I-O psychology graduate students [will] decide to enroll in business schools instead” (Aguinis et al., Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 548) is an inferential leap of major proportions and lends itself to alternate interpretations. Business schools provide a core of knowledge that focuses on strategy at the top of the totem pole (though I for one wonder why when those in strategy have apparently not solved the riddle of what preordains business success or failure), finance, accounting, marketing, management, and human resources (much lower on the totem pole), including labor relations, compensation, and deployment. Doctoral-level training prepares graduates virtually exclusively for academic positions.
On the other hand, I-O psychology rests on a core of human development and cognition, motivation and emotion, neuroscience, psychological aspects of leadership, social and societal influences, differential psychology (i.e., study of individual differences, which masquerades these days as talent management), human communication, and group or team dynamics. As for methodology, I-O psychology graduates bring sophistication in such areas as laboratory research and psychometrics. All these sources of knowledge are clearly of benefit to business schools as Kahneman and Tversky have shown so convincingly (e.g., Lewis, Reference Lewis2017). Doctoral graduates are prepared for and may enter a diversity of fields, not just academics, including private and governmental organizations, consulting firms, research organizations, NGOs, and even their own businesses. Thus business schools should have a keen interest in keeping the pipeline from I-O psychology programs open. Students will continue to seek the kind of graduate training that suits their interests and career plans regardless of where a subset of I-O psychology graduates are affiliated. Indeed the fact that business schools are willing to pay premium rates to bring I-O psychologists on board is an added inducement to enroll in an I-O psychology program, not to mention the wider expanse of options available to them when they graduate.
One way to evaluate the sustainability of I-O psychology programs, which Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) suggest is threatened by the migration of I-O psychology to business schools, is to note whether substantial changes in numbers and quality of applicants to I-O programs have occurred. The program I am most familiar with at the University of South Florida (USF) has attracted the following numbers of applicants over the past 5 years, starting in 2013, for four to six entrants each year: 121, 175, 151, 136, 108. These data suggest no lack of substantial competitiveness in the program across all years, and those entering the program are consistently highly talented. Dr. Tammy Allen, the I-O psychology program director, reports that the overall quality of the applicant pool remains high. The declining numbers over the past 3 years, however, might be the beginning of a trend indicative of declining interest. Complicating any firm conclusions for USF's program are other factors such as changes in the deadline date for applying, which has become much earlier than in the past.
Dr. Clint Bowers, chairperson of the Psychology Department at the University of Central Florida, has graciously provided data on both its doctoral and terminal master's program applicant pools. Over 5 years beginning in 2013, the data are as follows with master's program applicant numbers in parentheses and respective overall GPAs following each applicant total: 100, GPA 3.87 (119, GPA 3.56); 102, GPA 3.6 (109, GPA 3.74); 108, GPA 3.66 (135, GPA 3.59); 90, GPA 3.65 (159, GPA 3.8); 108, GPA 3.79 (128, GPA 3.77). Entrants ranged from four to seven for the doctoral program and 13 to 15 for the master's program. No consistent downward trend in the number or quality of the applicant pool is evident.
By contrast with these two large public universities the Florida Institute of Technology (FIT) is a much smaller private institution. It also offers a terminal master's program in addition to the doctoral program as UCF does. FIT's number of applicants, courtesy of Dr. Lisa Steelman, the program director, over 5 years starting in 2013 are as follows, with master's program applicant numbers in parentheses: 57 (76), 58 (58), 73 (92), 55 (80), 54 (66) with 5 to 10 doctoral program entrants and 5 to 11 master's program entrants each year. Dr. Steelman reports that GREs and GPAs have been relatively stable over this period. Again as with USF, the recent decline in applicant numbers creates questions.
Overall these data, albeit quite preliminary, do not support any imminent existential threat to the sustainability of these I-O psychology programs, but they do suggest that the recent declines at USF and FIT need to be tracked in future. Perhaps the improvement of the world economy over the past few years with a concomitant increase in job opportunities is a more potent contributor to any observed decline in applicant numbers than the migration of I-O psychology to business schools. Of course a larger and more representative sample of programs nationally and even internationally would be a far better gauge of any threat to I-O psychology programs’ viability.
I do not doubt, however, that staffing I-O psychology programs while maintaining internal pay equity within a psychology department will continue to be a serious challenge. Perhaps, ironically, business schools, given the benefits of I-O psychology programs to business schools, will find a way to provide support to I-O psychology programs in some way as a means of responding to this challenge. For example they might provide part time appointments to I-O psychology faculty housed in psychology departments to supplement the lower salaries paid in those departments.
Frequency of Author Citations in Textbooks Through the Lenses of Wallace's (Reference Wallace1965) “Criteria for What” and Weitz's (Reference Weitz1961) “Criteria for Criteria”
Most pertinent to the issue at hand, namely the use of frequency of author citations in I-O texts, Wallace (Reference Wallace1965), whose argumentation is based within the psychometric tradition of the I-O field, railed against choosing criteria for correlational, particularly predictive, studies on the basis of expediency and practicality without consideration of a deeper conceptual basis. He answered his question by asserting that choosing criteria to facilitate understanding was the key if I-O was to assume its place as a true scientific discipline. Construct validity was seen as crucial to achieve this objective.
This search for understanding suggests that Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) have done a great service by advancing the question of the meaning of impact. However, it is clear that much more remains to be done to elucidate an understanding of the construct of impact and how authors’ citation counts in I-O psychology introductory texts factor in.
From a practical perspective, this lack of understanding also precludes successful performance management in the domain of academia. If we follow good practice in goal setting, we would specify for faculty quantity, quality, process requirements, deadlines, and/or cost considerations. This contrasts with my usual pronouncements to new faculty, for example when I served as chairperson of my department, namely, publish some articles in rigorous journals (not sure how many, not sure if textbook writing is valued); bring in some grant money; do not get slammed by student course ratings; make friends; and help out around the department, but not too much. This should lead to tenure and promotion. If I were to suggest that they pursue getting frequent citations in introductory I-O psychology textbooks, there is no clear path to achieve this. Certainly as Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) show, there is a moderate correlation between Web of Science citations and textbook citations, and authoring meta-analyses gives one about a 17% chance of cracking the most cited articles (84 articles among the 219 most cited were meta-analyses). But the path to success on this criterion is far from clear. (I suspect that if this becomes a performance requirement, I-O psychology textbook authors can count on tributes of various kinds, flowers and candy among them.)
Weitz (Reference Weitz1961) writes out of I-O psychology's experimental laboratory tradition, which was a bedrock in the emergence of I-O psychology as a science owing to the work of such pioneers as Hugo Munsterberg. Weitz identified three criteria for criteria: time, type, and level.
Time refers to how long after a manipulation of independent variables do we take our criterion measure. Type is the construct to be captured by the criterion. Level is what standard of performance must be attained to reach a desired outcome, such as mastery in a training context.
In the case of citation frequency we have substantial problems. Time after completion of graduate studies to begin counting citations would correspond to Weitz's time criterion. The frequency count used by Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) does not equate time in career across authors, nor does it sample across time periods for the count. The former metric would perhaps bias counts from textbooks in favor of older authors who have had more time to publish, whereas the latter would favor younger authors when textbooks are under pressure to cite new material. Using texts from different eras might for example place on the list luminaries such as Cronbach, Meehl, and Herzberg (who served as a very popular starting point for critiques and research in I-O psychology in his day), and might elevate luminaries like Guion to a much higher place on the list.
As for type, once again the issue of construct validity of the measure rises. Based on correlational data, there is some indication of overlap with research citations, but the data and theory on construct validity are sparse or lacking. We need to drill much deeper into the meaning of impact and likely explore multimethod ways to get at indices. For example, I have no doubt that Jacob Cohen was and is highly impactful by virtue of his work on multiple regression and analysis of variance (ANOVA) as two sides of the same coin, statistical power, frailties of null hypothesis testing, and kappa, but he does not appear on the list of the top 178. This is because his work is foundational for the research of others, but his articles do not get cited in I-O psychology textbooks.
Level poses great difficulty also in that merely having more citations than others does not inform about how much impact has been achieved; rather it merely indicates that one has more or less than others. Returning to the hypothetical example of counseling young faculty, a chair might say to the new faculty member that he/she needs to get more than similarly situated others, but how many are needed to get desired outcomes like tenure and promotion, or what constitutes excellence, cannot be specified.
Overall in my view Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) deserve much credit for pushing higher on the research agenda the question of impact, which incidentally deserves to be evaluated for I-O psychology practitioners as well as academics. Furthermore, they have provided a rich and carefully constructed data set to help advance the study of impact. Hopefully they and others will continue to pursue the resolution to this form of the eternal criterion problem.