Academics sometimes forget that the purpose of a university is to educate: our students, our local communities, each other, and the world. Although each university is unique in its constituency, all share the charge to generate knowledge for the protection and benefit of the public good. The goal of an academic should be to beneficially impact society, broadly defined, with scholarly activity. As editor and columnist for The Industrial-Organizational Psychologist, one publication highlighted by the focal article, we applaud the efforts of Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) to put forth alternative approaches to defining impact. Like them, we are concerned that many of the measures of “impact” we currently use do not capture this charge.
The scholarly world is changing. Universities respond to their environments, and the current environment is one of quantification, standardization, and competition for resources. If universities decide to hold their faculty to rigorous quantitative standards in this harsh reality, those standards have to come from somewhere. Some products of scholarship are easy to count, such as number of journal publications and the impact factor of those publications. Others are more difficult to count: students inspired, minds changed, worker lives improved. Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) attempt to inject broader variety into what we consider valuable scholarship. Their attempt does not need to be perfect in order to drive our understanding forward, as it makes us question why we place value on the things we do.
Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 547) ask in their focal article, “Should tenure and promotion systems [. . .] expand their journal lists to include bridge journals such as TIP, Human Resource Management, and Organizational Dynamics, which do not enjoy particularly high JCR impact factor scores (or do not have an impact factor score at all) but are nevertheless cited frequently in textbooks (ranked #14, #35, and #47, respectively)?” We are not sure about the answer to this question. In some respects, no expanded measure will ever be adequate, because once a measure is used for goal setting, its construct validity as an indicator of success is harmed. Economists refer to this problem as Goodhart's law: Any measure, once used as a target, ceases to be a good measure. Put another way, once journal impact factors become a hurdle for career progress or profit, authors, editors, and publishers will do what they have to in order to maximize their impact factors. This in turn often leads people to find ways to game the system to maximize their outcomes, making those numbers less meaningful. The assignment and emphasis of impact factors is therefore a deeply political process. Critics of standardized testing in education complain about this problem in another way when they bemoan “teaching to the test.” In I-O psychology, we are familiar with this problem as a possible dark side of goal setting (Ordonez, Schweitzer, Galinsky, & Bazerman, Reference Ordonez, Schweitzer, Galinsky and Bazerman2009).
Extant alternative metrics of impact such as the Altmetrics effort, which is an attempt to quantify impact based on penetration into popular media outlets, do not suffice. Based on “buzz” such as retweets and news coverage, this sort of metric rewards flashiness over substance. We should not be obsessed with novelty, but it remains unclear what we should be obsessed with. There are even alternatives to the alternatives—for example, if a piece of research is cited in a policy document, it has the potential to affect the lives of thousands or even millions of people. Research on selection has been enormously influential in determining Equal Employment Opportunity (EEO) law. Shouldn't this “count” at least as much as an article cited once or twice in the scholarly literature?
Perhaps the best summary of our impact challenge comes from a landmark paper—cited over 10,000 times—by Rittel and Webber (Reference Rittel and Webber1973) defining wicked problems. These are problems that are difficult to define and can never truly be “solved,” with shades of gray in both. They are problems where the solutions cannot truly be tested; we cannot learn by trial and error or experimentation because the stakes are so high. There is no limit to the number of potential solutions, and every potential solution is likely to lead to multiple new challenges yet unidentified. Impact is a wicked problem. It cannot be easily escaped. If comparisons between researchers are necessary, then some measurement of performance is needed, but the myriad stakeholders involved with highly discrepant goals means that no single operationalization will ever be satisfactory to all. We cannot even experiment with alternative systems to see their effects without damaging careers in the process, because many careers have been optimized on a flawed system.
In determining the best measures of impact, we—as a field or likely failing that, as individuals—must first decide what the goal of scholarship should be. It is only with specific goals in mind that we can accurately develop measures that capture progress towards those goals (Locke & Latham, Reference Locke and Latham1990). We (the authors) spend a lot of time thinking about the goals of scholarship as they relate to our work for TIP. One important feature of TIP is that it is open access. Anyone can read it. Based on Google Analytics data, the most popular articles in TIP are read by hundreds (and sometimes thousands) of people each month. This is a direct form of impact that many peer-reviewed academic journals can only hope for—as much as we believe that our top journal publications are valuable, we know that we've hit the scholarly jackpot if even a few dozen authors cite them. Recently, the Gates Foundation made a major effort to improve the state of open access by requiring that all recipients of Gates funding must publish their work in open access journals, stating their belief in transparency and immediate dissemination of findings as core goals of research (Gates Foundation, 2015). This is a promising move and is consistent with the logic that everyone should be able to benefit from the findings of scholarly research.
Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017), in writing about outlets such as TIP, note that “these sources speak to and feature contributions from both academics and practitioners. Although these journals do not have a high JCR impact factor, they nevertheless feature prominently in introductory I-O psychology textbooks. Thus, these journals represent an outlet that academics can use to bridge the science–practice divide and impact both current and future practitioners” (p. 546). We agree. We encourage our peers to reflect on the true purpose of scholarship and to worry less about gaming the very narrowly defined set of impact metrics that are currently relied upon by default. Impact factors are an interesting piece of information—one part of the impact puzzle—but not the most important and certainly not an appropriate job goal. As argued by Verma (Reference Verma2015; in the high-impact-factor journal Proceedings of the National Academy of Sciences), think of impact, not impact factors.
You have
Access
Academics sometimes forget that the purpose of a university is to educate: our students, our local communities, each other, and the world. Although each university is unique in its constituency, all share the charge to generate knowledge for the protection and benefit of the public good. The goal of an academic should be to beneficially impact society, broadly defined, with scholarly activity. As editor and columnist for The Industrial-Organizational Psychologist, one publication highlighted by the focal article, we applaud the efforts of Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) to put forth alternative approaches to defining impact. Like them, we are concerned that many of the measures of “impact” we currently use do not capture this charge.
The scholarly world is changing. Universities respond to their environments, and the current environment is one of quantification, standardization, and competition for resources. If universities decide to hold their faculty to rigorous quantitative standards in this harsh reality, those standards have to come from somewhere. Some products of scholarship are easy to count, such as number of journal publications and the impact factor of those publications. Others are more difficult to count: students inspired, minds changed, worker lives improved. Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017) attempt to inject broader variety into what we consider valuable scholarship. Their attempt does not need to be perfect in order to drive our understanding forward, as it makes us question why we place value on the things we do.
Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017, p. 547) ask in their focal article, “Should tenure and promotion systems [. . .] expand their journal lists to include bridge journals such as TIP, Human Resource Management, and Organizational Dynamics, which do not enjoy particularly high JCR impact factor scores (or do not have an impact factor score at all) but are nevertheless cited frequently in textbooks (ranked #14, #35, and #47, respectively)?” We are not sure about the answer to this question. In some respects, no expanded measure will ever be adequate, because once a measure is used for goal setting, its construct validity as an indicator of success is harmed. Economists refer to this problem as Goodhart's law: Any measure, once used as a target, ceases to be a good measure. Put another way, once journal impact factors become a hurdle for career progress or profit, authors, editors, and publishers will do what they have to in order to maximize their impact factors. This in turn often leads people to find ways to game the system to maximize their outcomes, making those numbers less meaningful. The assignment and emphasis of impact factors is therefore a deeply political process. Critics of standardized testing in education complain about this problem in another way when they bemoan “teaching to the test.” In I-O psychology, we are familiar with this problem as a possible dark side of goal setting (Ordonez, Schweitzer, Galinsky, & Bazerman, Reference Ordonez, Schweitzer, Galinsky and Bazerman2009).
Extant alternative metrics of impact such as the Altmetrics effort, which is an attempt to quantify impact based on penetration into popular media outlets, do not suffice. Based on “buzz” such as retweets and news coverage, this sort of metric rewards flashiness over substance. We should not be obsessed with novelty, but it remains unclear what we should be obsessed with. There are even alternatives to the alternatives—for example, if a piece of research is cited in a policy document, it has the potential to affect the lives of thousands or even millions of people. Research on selection has been enormously influential in determining Equal Employment Opportunity (EEO) law. Shouldn't this “count” at least as much as an article cited once or twice in the scholarly literature?
Perhaps the best summary of our impact challenge comes from a landmark paper—cited over 10,000 times—by Rittel and Webber (Reference Rittel and Webber1973) defining wicked problems. These are problems that are difficult to define and can never truly be “solved,” with shades of gray in both. They are problems where the solutions cannot truly be tested; we cannot learn by trial and error or experimentation because the stakes are so high. There is no limit to the number of potential solutions, and every potential solution is likely to lead to multiple new challenges yet unidentified. Impact is a wicked problem. It cannot be easily escaped. If comparisons between researchers are necessary, then some measurement of performance is needed, but the myriad stakeholders involved with highly discrepant goals means that no single operationalization will ever be satisfactory to all. We cannot even experiment with alternative systems to see their effects without damaging careers in the process, because many careers have been optimized on a flawed system.
In determining the best measures of impact, we—as a field or likely failing that, as individuals—must first decide what the goal of scholarship should be. It is only with specific goals in mind that we can accurately develop measures that capture progress towards those goals (Locke & Latham, Reference Locke and Latham1990). We (the authors) spend a lot of time thinking about the goals of scholarship as they relate to our work for TIP. One important feature of TIP is that it is open access. Anyone can read it. Based on Google Analytics data, the most popular articles in TIP are read by hundreds (and sometimes thousands) of people each month. This is a direct form of impact that many peer-reviewed academic journals can only hope for—as much as we believe that our top journal publications are valuable, we know that we've hit the scholarly jackpot if even a few dozen authors cite them. Recently, the Gates Foundation made a major effort to improve the state of open access by requiring that all recipients of Gates funding must publish their work in open access journals, stating their belief in transparency and immediate dissemination of findings as core goals of research (Gates Foundation, 2015). This is a promising move and is consistent with the logic that everyone should be able to benefit from the findings of scholarly research.
Aguinis et al. (Reference Aguinis, Ramani, Campbell, Bernal-Turnes, Drewry and Edgerton2017), in writing about outlets such as TIP, note that “these sources speak to and feature contributions from both academics and practitioners. Although these journals do not have a high JCR impact factor, they nevertheless feature prominently in introductory I-O psychology textbooks. Thus, these journals represent an outlet that academics can use to bridge the science–practice divide and impact both current and future practitioners” (p. 546). We agree. We encourage our peers to reflect on the true purpose of scholarship and to worry less about gaming the very narrowly defined set of impact metrics that are currently relied upon by default. Impact factors are an interesting piece of information—one part of the impact puzzle—but not the most important and certainly not an appropriate job goal. As argued by Verma (Reference Verma2015; in the high-impact-factor journal Proceedings of the National Academy of Sciences), think of impact, not impact factors.