In their focal article, Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) categorized the 6,654 unique citations, summed across the six introductory industrial and organizational (I-O) psychology texts, in various ways. They then suggested how such data could be used to (a) infer the “state” of the scientist–practitioner divide; (b) document the extent of the movement of I-O psychologists to management schools; (c) evaluate the future prospects of I-O psychology as a field; and (d) provide guidance in how to define, measure, and reward “scholarly impact” (quotation marks added). This crosses the line from interesting to very counterproductive.
Some Context
Their suggestions for how to use these data must be evaluated within a very multifaceted context. Here, I wish to address only two such facets. The first is that as a domain for theory, research, and practice, the world of work is fundamental and important. Its importance cannot be overstated. It is a major part of most people's lives. Without “work,” human civilization simply disappears. Consequently, even though we may or may not have identity problems, and we can argue about whether we are better housed in psychology or management or both, our field is important for the survival of just about everybody. The need for evidenced-based practice looms large. Our field must grow in importance, by orders of magnitude. However, if we are not careful, and perhaps contrary to many sciences, I-O psychology can easily be replaced by “common sense” and the idiosyncratic personal beliefs of people in positions of responsibility. We should all keep that in mind and not be too distracted by identity issues. Neither should we adopt self-defeating practices.
The second critical contextual feature consists of the set of possible goals toward which theory, research, and practice (TRP) can be directed. Campbell and Wilmot (Reference Campbell, Wilmot, Anderson, Ones, Sinangil and Viswesvaran2018) argue that, in our business, there are five goals. A summary of the five is as follows.
-
Goal One. Improve organizational effectiveness, realizing that there are many ways to index effectiveness, and that the effectiveness indicators would vary depending on the kind of organization being considered (e.g., profit making, NGO, governmental, educational, etc.). That is, financial return to shareholders is not the only measure of effectiveness, although the general goal of “increasing the return to stakeholders” applies to virtually any organization. A governmental organization could think of what it returns to its constituencies and then argue about the best indicators of such returns (e.g., minimizing the tax burden for the 1% or providing equal educational opportunities for all). An excellent decomposition of what organizational effectiveness means is provided by Chandler (Reference Chandler2015). His principal point is that effectiveness refers to the degree to which customers, clients, or other stakeholders actually use the external outcomes produced by the organization.
-
Goal Two. Improve the well-being of individuals in work roles as judged from the perspective of the individual, not the organization or the science. The usual indicators of such well-being are levels of job and life satisfaction, engagement, work–family conflict, stress level, promotional opportunities, and individual skill development. From the individual perspective, these indicators are the dependent variables. In this context, they are not viewed as independent variables that “cause” increases in organizational effectiveness, although that may be an added benefit.
-
Goal Three. Contribute to the “common good.” That is, beyond a specific organization's effectiveness and beyond the well-being of specific individuals, but still very much within the world of work, there are societal “common goods” to be served, such as reducing conflict, improving health care, improving education, reducing poverty, improving environmental sustainability, improving governance, and promoting achievement and innovation. The 2012 Academy of Management presidential address by Anne Tsui (Reference Tsui2013) spoke eloquently about goals for the common good.
-
Goal Four. The goal of our theory, research, and practice could be just to “learn stuff” that is, or might be, important for one or more of the first three goals. Any science or profession should support some degree of TRP that has no specific goal other than to better understand phenomena of interest that are related to the world of work and organizations. Call this basic research if you want, but it should be part of any discipline, and it should be supported, not shunned.
-
Goal Five. Finally, the goal of theory, research, and practice could be to produce frequently cited articles in “top-tier” journals, such that the number of publications, and their attendant citation rates, are sufficient to yield job opportunities, tenure, salary increases, promotions, and other rewards. Such “big data” have also become the primary indicator of the worth, or value, of an individual's contribution to the science itself (e.g., Lawrence, Reference Lawrence2003). The content of the publications, and their contributions to any subset of the first four goals, is of secondary importance. As argued below, the power of this goal is degrading our field, perhaps to a state of irrelevance, sooner rather than later.
Goal Five has already done much damage. Campbell and Wilmot (Reference Campbell, Wilmot, Anderson, Ones, Sinangil and Viswesvaran2018) have summarized the destructive results of pursuing Goal Five, as seen by many other commentators (e.g., Fanelli, Reference Fanelli2009; Ferguson & Heene, 2012; Honig, Lampel, Siegel, & Drnevich, Reference Honig, Lampel, Siegel and Drnevich2014; Ilgen, Reference Ilgen2007; Ioannidis, Reference Ioannidis2011; Lawrence, Reference Lawrence2008; Nosek, Spies, & Motyl, Reference Nosek, Spies and Motyl2012). A brief summary of their summary is as follows.
Among other things, Goal Five promotes the following. Call them methods for gaming the system.
-
• Divide a program of research into as many publishable parts as possible.
-
• Do not submit replications. Few, if any, are accepted.
-
• Submit papers on new or novel theories for theory's sake (TFTSs), not because they point the way to more useful research that addresses important questions, but because new “theory” is valued for its own sake.
-
• Only submit empirical papers that show positive (i.e., statistically significant) results.
-
• Use a variety of questionable research and analysis practices (QRAPs) to increase the probability of positive results.
-
• Do not submit papers that yield negative or nil results.
-
• Submit papers that deal with “hot” topics, as determined from the topics that a journal publishes frequently. Pay particular attention to “tier-one” journals.
-
• Do not engage interesting, important, or critical issues if it will require large or long-term time commitments that will delay being able to submit manuscripts for publication.
-
• Do not try to address the science–practice divide. It only lowers publication frequency and citation counts.
-
• Have multiple, and I do mean multiple, coauthors, even if some of them really have had little or nothing to do with producing the paper. It increases everyone's publication count.
-
• Maximize self-citations, and urge all coauthors to do likewise.
-
• Develop networks of other researchers and theorists who explicitly, or implicitly, agree to cite other members of the network as much as possible, and cite outside the network as little as possible, regardless of substantive appropriateness.
Although outcomes such as number of publications and citation counts should bear some relationship to the excellence of substantive performance itself, the gaming strategies introduce much bias.
Goal Five and Textbook Citations
Now, what does Goal Five have to do with the patterns, frequencies, and sources of the citations found in introductory I-O psychology textbooks? The primary consideration is that Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 547) state that, “Our findings also lead to implications for the design of faculty performance management systems.” They make the point that there is considerable overlap between the authors who have high citation rates in general and the authors who are cited most often in the textbooks. Consequently, they are advocating that certain citation rates gleaned from big data analyses of textbooks can be used to evaluate the performance of faculty. That is, Goal Five prevails. Elsewhere, Aguinis, Shapiro, Antonacopoulou, and Cummings (Reference Aguinis, Shapiro, Antonacopoulou and Cummings2014) have recognized the susceptibility of citation rates to various biases and advocate a more “pluralistic” approach to using citations and publication counts, as well as other “counts” for assessing faculty performance. See their table 2. It includes such things as number of practitioner publications; number of popular press books; number of likes, clicks, comments, shares, and tweets on Facebook, Google, Twitter, academia.edu, and Research Gate; and the number of bookmarks based on Delicious, Mendeley, Reddit, and so on. However, using more specific and more varied categories of things to count, for example as found in social media, or in what are now called “altmetrics,” is still subject to the Goal Five effects discussed above and does not assess the actual substantive performance of individuals. It would most certainly multiply the depth and breadth of Goal Five gaming strategies.
Previously, Aguinis and O'Boyle (Reference Aguinis, Shapiro, Antonacopoulou and Cummings2014) and O'Boyle and Aguinis (Reference O'Boyle and Aguinis2012) have argued that I-O psychology should not attempt to assess actual individual performance directly but should rely on assessing the outcomes of performance, because direct assessments (usually ratings) are too biased and unreliable. Essentially, this is an argument that Goal Five should be our most important goal because the degree to which an individual excels on Goal Five can be “measured.” By implication, excelling on Goal Five means excelling on one or more of the first four goals as well. As cited above, many have argued that this is simply not true. Goal Five is counterproductive.
A very recent, strongly worded statement of the anti–Goal Five argument is given in Edwards and Roy (Reference Edwards and Roy2017). They write from the perspective of environmental science and engineering, but the issues they raise are the same. In their view, Goal Five lowers research quality, detracts from answering important questions, leads to unethical behavior, debases scientific integrity, and alters the values of the university itself. In their view, we are creating a corrupt academic culture, and, if nothing is done, all of science will fall into disrepute. Certain political factions will have a field day.
Some Conclusions
Although some might find the citation counts presented in the focal article interesting, they are simply counterproductive. They reinforce the notion that substantive judgments about individual performance and impact should take a distant backseat to big data counts. That is, publications should be counted, not read (MacDonald & Kim, Reference MacDonald and Kam2007). The actual substantive contributions of researchers and practitioners are not important.
With regard to the content of I-O psychology textbooks, would it not have been better to develop taxonomies of actual content, including the research and practice sources of the content, such that the nature of the substantive content could be compared across texts and across time periods, as well as making other comparisons that might be interesting? For example, are there important areas of TRP that are not represented in the texts? If so, might that be for good reasons or not so good reasons? As reflected in the texts, what do we actually currently know about promoting Goals One, Two, Three, and Four?
These issues also confront the I-O psychology enterprise at large. That is, what are the critical substantive questions that our field should address, both now and in the future? How well are individual theorists, researchers, and practitioners addressing them? Are we ceding too much to the economists, sociologists, social psychologists, and other disciplines? Relatedly, what processes can/should we use to identify the substantive questions and issues most critical for achieving Goals One through Four, realizing that there is no one critical “list.” Also, the processes for identifying critical issues must be dynamic, multifaceted, and continuously ongoing for as long as I-O psychology exists. That is, continuously focusing in various ways on the substance of our research and practice needs should be at the bedrock of our culture. Appealing to citation counts is no answer.
Unfortunately, Goal Five is still the most important goal for theory, research, and practice in I-O psychology, at least for academics. This allegiance to Goal Five inhibits contributions to Goals One through Four, perpetuates the science–practice gap, and leads to much QRAPing and gaming. It will continue to degrade the science to a greater and greater extent. Let's get rid of it, and attempt to assess individual researcher/theorist substantive performance itself, difficult and time consuming though that is. The viability of our field depends on it.
You have
Access
In their focal article, Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) categorized the 6,654 unique citations, summed across the six introductory industrial and organizational (I-O) psychology texts, in various ways. They then suggested how such data could be used to (a) infer the “state” of the scientist–practitioner divide; (b) document the extent of the movement of I-O psychologists to management schools; (c) evaluate the future prospects of I-O psychology as a field; and (d) provide guidance in how to define, measure, and reward “scholarly impact” (quotation marks added). This crosses the line from interesting to very counterproductive.
Some Context
Their suggestions for how to use these data must be evaluated within a very multifaceted context. Here, I wish to address only two such facets. The first is that as a domain for theory, research, and practice, the world of work is fundamental and important. Its importance cannot be overstated. It is a major part of most people's lives. Without “work,” human civilization simply disappears. Consequently, even though we may or may not have identity problems, and we can argue about whether we are better housed in psychology or management or both, our field is important for the survival of just about everybody. The need for evidenced-based practice looms large. Our field must grow in importance, by orders of magnitude. However, if we are not careful, and perhaps contrary to many sciences, I-O psychology can easily be replaced by “common sense” and the idiosyncratic personal beliefs of people in positions of responsibility. We should all keep that in mind and not be too distracted by identity issues. Neither should we adopt self-defeating practices.
The second critical contextual feature consists of the set of possible goals toward which theory, research, and practice (TRP) can be directed. Campbell and Wilmot (Reference Campbell, Wilmot, Anderson, Ones, Sinangil and Viswesvaran2018) argue that, in our business, there are five goals. A summary of the five is as follows.
Goal One. Improve organizational effectiveness, realizing that there are many ways to index effectiveness, and that the effectiveness indicators would vary depending on the kind of organization being considered (e.g., profit making, NGO, governmental, educational, etc.). That is, financial return to shareholders is not the only measure of effectiveness, although the general goal of “increasing the return to stakeholders” applies to virtually any organization. A governmental organization could think of what it returns to its constituencies and then argue about the best indicators of such returns (e.g., minimizing the tax burden for the 1% or providing equal educational opportunities for all). An excellent decomposition of what organizational effectiveness means is provided by Chandler (Reference Chandler2015). His principal point is that effectiveness refers to the degree to which customers, clients, or other stakeholders actually use the external outcomes produced by the organization.
Goal Two. Improve the well-being of individuals in work roles as judged from the perspective of the individual, not the organization or the science. The usual indicators of such well-being are levels of job and life satisfaction, engagement, work–family conflict, stress level, promotional opportunities, and individual skill development. From the individual perspective, these indicators are the dependent variables. In this context, they are not viewed as independent variables that “cause” increases in organizational effectiveness, although that may be an added benefit.
Goal Three. Contribute to the “common good.” That is, beyond a specific organization's effectiveness and beyond the well-being of specific individuals, but still very much within the world of work, there are societal “common goods” to be served, such as reducing conflict, improving health care, improving education, reducing poverty, improving environmental sustainability, improving governance, and promoting achievement and innovation. The 2012 Academy of Management presidential address by Anne Tsui (Reference Tsui2013) spoke eloquently about goals for the common good.
Goal Four. The goal of our theory, research, and practice could be just to “learn stuff” that is, or might be, important for one or more of the first three goals. Any science or profession should support some degree of TRP that has no specific goal other than to better understand phenomena of interest that are related to the world of work and organizations. Call this basic research if you want, but it should be part of any discipline, and it should be supported, not shunned.
Goal Five. Finally, the goal of theory, research, and practice could be to produce frequently cited articles in “top-tier” journals, such that the number of publications, and their attendant citation rates, are sufficient to yield job opportunities, tenure, salary increases, promotions, and other rewards. Such “big data” have also become the primary indicator of the worth, or value, of an individual's contribution to the science itself (e.g., Lawrence, Reference Lawrence2003). The content of the publications, and their contributions to any subset of the first four goals, is of secondary importance. As argued below, the power of this goal is degrading our field, perhaps to a state of irrelevance, sooner rather than later.
Goal Five has already done much damage. Campbell and Wilmot (Reference Campbell, Wilmot, Anderson, Ones, Sinangil and Viswesvaran2018) have summarized the destructive results of pursuing Goal Five, as seen by many other commentators (e.g., Fanelli, Reference Fanelli2009; Ferguson & Heene, 2012; Honig, Lampel, Siegel, & Drnevich, Reference Honig, Lampel, Siegel and Drnevich2014; Ilgen, Reference Ilgen2007; Ioannidis, Reference Ioannidis2011; Lawrence, Reference Lawrence2008; Nosek, Spies, & Motyl, Reference Nosek, Spies and Motyl2012). A brief summary of their summary is as follows.
Among other things, Goal Five promotes the following. Call them methods for gaming the system.
• Divide a program of research into as many publishable parts as possible.
• Do not submit replications. Few, if any, are accepted.
• Submit papers on new or novel theories for theory's sake (TFTSs), not because they point the way to more useful research that addresses important questions, but because new “theory” is valued for its own sake.
• Only submit empirical papers that show positive (i.e., statistically significant) results.
• Use a variety of questionable research and analysis practices (QRAPs) to increase the probability of positive results.
• Do not submit papers that yield negative or nil results.
• Submit papers that deal with “hot” topics, as determined from the topics that a journal publishes frequently. Pay particular attention to “tier-one” journals.
• Do not engage interesting, important, or critical issues if it will require large or long-term time commitments that will delay being able to submit manuscripts for publication.
• Do not try to address the science–practice divide. It only lowers publication frequency and citation counts.
• Have multiple, and I do mean multiple, coauthors, even if some of them really have had little or nothing to do with producing the paper. It increases everyone's publication count.
• Maximize self-citations, and urge all coauthors to do likewise.
• Develop networks of other researchers and theorists who explicitly, or implicitly, agree to cite other members of the network as much as possible, and cite outside the network as little as possible, regardless of substantive appropriateness.
Although outcomes such as number of publications and citation counts should bear some relationship to the excellence of substantive performance itself, the gaming strategies introduce much bias.
Goal Five and Textbook Citations
Now, what does Goal Five have to do with the patterns, frequencies, and sources of the citations found in introductory I-O psychology textbooks? The primary consideration is that Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 547) state that, “Our findings also lead to implications for the design of faculty performance management systems.” They make the point that there is considerable overlap between the authors who have high citation rates in general and the authors who are cited most often in the textbooks. Consequently, they are advocating that certain citation rates gleaned from big data analyses of textbooks can be used to evaluate the performance of faculty. That is, Goal Five prevails. Elsewhere, Aguinis, Shapiro, Antonacopoulou, and Cummings (Reference Aguinis, Shapiro, Antonacopoulou and Cummings2014) have recognized the susceptibility of citation rates to various biases and advocate a more “pluralistic” approach to using citations and publication counts, as well as other “counts” for assessing faculty performance. See their table 2. It includes such things as number of practitioner publications; number of popular press books; number of likes, clicks, comments, shares, and tweets on Facebook, Google, Twitter, academia.edu, and Research Gate; and the number of bookmarks based on Delicious, Mendeley, Reddit, and so on. However, using more specific and more varied categories of things to count, for example as found in social media, or in what are now called “altmetrics,” is still subject to the Goal Five effects discussed above and does not assess the actual substantive performance of individuals. It would most certainly multiply the depth and breadth of Goal Five gaming strategies.
Previously, Aguinis and O'Boyle (Reference Aguinis, Shapiro, Antonacopoulou and Cummings2014) and O'Boyle and Aguinis (Reference O'Boyle and Aguinis2012) have argued that I-O psychology should not attempt to assess actual individual performance directly but should rely on assessing the outcomes of performance, because direct assessments (usually ratings) are too biased and unreliable. Essentially, this is an argument that Goal Five should be our most important goal because the degree to which an individual excels on Goal Five can be “measured.” By implication, excelling on Goal Five means excelling on one or more of the first four goals as well. As cited above, many have argued that this is simply not true. Goal Five is counterproductive.
A very recent, strongly worded statement of the anti–Goal Five argument is given in Edwards and Roy (Reference Edwards and Roy2017). They write from the perspective of environmental science and engineering, but the issues they raise are the same. In their view, Goal Five lowers research quality, detracts from answering important questions, leads to unethical behavior, debases scientific integrity, and alters the values of the university itself. In their view, we are creating a corrupt academic culture, and, if nothing is done, all of science will fall into disrepute. Certain political factions will have a field day.
Some Conclusions
Although some might find the citation counts presented in the focal article interesting, they are simply counterproductive. They reinforce the notion that substantive judgments about individual performance and impact should take a distant backseat to big data counts. That is, publications should be counted, not read (MacDonald & Kim, Reference MacDonald and Kam2007). The actual substantive contributions of researchers and practitioners are not important.
With regard to the content of I-O psychology textbooks, would it not have been better to develop taxonomies of actual content, including the research and practice sources of the content, such that the nature of the substantive content could be compared across texts and across time periods, as well as making other comparisons that might be interesting? For example, are there important areas of TRP that are not represented in the texts? If so, might that be for good reasons or not so good reasons? As reflected in the texts, what do we actually currently know about promoting Goals One, Two, Three, and Four?
These issues also confront the I-O psychology enterprise at large. That is, what are the critical substantive questions that our field should address, both now and in the future? How well are individual theorists, researchers, and practitioners addressing them? Are we ceding too much to the economists, sociologists, social psychologists, and other disciplines? Relatedly, what processes can/should we use to identify the substantive questions and issues most critical for achieving Goals One through Four, realizing that there is no one critical “list.” Also, the processes for identifying critical issues must be dynamic, multifaceted, and continuously ongoing for as long as I-O psychology exists. That is, continuously focusing in various ways on the substance of our research and practice needs should be at the bedrock of our culture. Appealing to citation counts is no answer.
Unfortunately, Goal Five is still the most important goal for theory, research, and practice in I-O psychology, at least for academics. This allegiance to Goal Five inhibits contributions to Goals One through Four, perpetuates the science–practice gap, and leads to much QRAPing and gaming. It will continue to degrade the science to a greater and greater extent. Let's get rid of it, and attempt to assess individual researcher/theorist substantive performance itself, difficult and time consuming though that is. The viability of our field depends on it.