The TOP factor

In our assessment of journal quality, we may have confused journal quality with popularity. So, what are the solutions? Perhaps an initial starting point is to simply acknowledge that the JIF is a measure of popularity and not necessarily journal quality. Ultimately, to assess quality, it would be a good starting point to measure how many articles within a journal conform to our very own reporting standards for journal articles (Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018; Levitt et al., Reference Levitt, Bamberg, Creswell, Frost, Josselson and Suárez-Orozco2018), which were devised to ensure that published articles provide some basic level of information to properly evaluate them (Cooper, Reference Cooper2020). Unfortunately, a quick glance at them illustrates that most published articles fall short of adhering to them (e.g., Schalken & Rietbergen, Reference Schalken and Rietbergen2017), which would mean that at least some of our journals would be considered poor quality (e.g., when was the last time an article in our most prestigious journals reported the recommended power analysis?). Similarly obvious is the fact that, without detailed descriptions of the research ingredients (e.g., theory and measures) and instructions (research design and analytical approach), it is impossible to properly replicate published studies and assess the quality of the articles in which they are published (Cooper, Reference Cooper2020). Thus, an assessment of the adherence of published articles to our own reporting standards is likely to be a more valuable quality assessment than metrics based on citation counts. Yet, such an assessment may take years to implement.

A good first step towards such an assessment could be the promotion and use of the recently released Transparency and Openness (TOP) factor (https://cos.io/about/news/new-measure-rates-quality-research-journals-policies-promote-transparency-and-reproducibility/), which comprises 10 distinct dimensions that assess the “quality of research journals’ policies to promote transparency and reproducibility” on scales that range from “0” (a journal’s policy fails to meet the standard) to “3” (a journal’s policy includes both a requirement and a verification process for the data’s correspondence with the findings reported in the paper), indicating the degree to which a journal adheres to best practices regarding transparency and reproducibility (Center for Open Science, 2020). The TOP guidelines were published a few years ago in Science and encourage the sharing of data and analytic code, study preregistration, and replications, among other things (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and Yarkoni2015). They now have over 5,000 journal signatories, and many major publishers endorse them as well. The TOP factor is meant to assess journals based on the quality of the implementation of their processes and their adherence to scholarly values as presented in the TOP guidelines. Although the 10 dimensions are not exhaustive (e.g., an assessment of a two-stage review process is not included; Kepes & McDaniel, Reference Kepes and McDaniel2013), they are quite comprehensive. Interestingly, the TOP factor is a rating rather than a ranking, which means that, hypothetically, all journals in I-O psychology and management can have a perfect score. Although a high TOP factor does not directly indicate that an article is necessarily of high quality, it does facilitate the evaluation of article quality.

Why is the TOP factor needed for I-O psychology and management journals? For years now, issues related to the credibility of our sciences (e.g., research integrity, data transparency, reproducibility, publication bias) seem to be one of the major challenges in I-O psychology, management, and other scientific fields (e.g., Kepes & McDaniel, Reference Kepes and McDaniel2013; National Academies of Sciences, 2017). Indeed, journals across the sciences seem to be taking these issues very seriously by, for instance, requiring the preregistration of studies and/or the submission of syntax and/or raw data as a requirement for publication (e.g., American Finance Association, n.d.; Dickersin & Rennie, Reference Dickersin and Rennie2012; Miyakawa, Reference Miyakawa2020; PLoS One, n.d.), which is exactly what the TOP factor assesses. As is becoming increasingly clear, some of the results published in our most prestigious journals are unlikely to replicate (e.g., Kepes et al., Reference Kepes, List and McDaniel2018; O’Boyle et al., Reference O’Boyle, Banks, Carter, Walter and Yuan2019; Saylors & Trafimow, Reference Saylors and Trafimowin press). Therefore, how good are our most prestigious journals from a perspective of scientific rigor, accuracy, and usefulness? What is their TOP factor?

Using the TOP factor, I-O psychology and management journals could, for instance, score a “0” on the “data transparency” dimension, meaning the journal does not have a data-sharing policy (i.e., there is no comment from the journal on the matter). A score of 1 indicates that a policy exists in which authors of articles have to report whether or not their data are available. A score of 2 for data sharing would indicate that authors are required to share data, but exceptions are allowed in cases of proprietary data or if there is a particular concern about participant identities. The highest score is a 3 in which there is both a policy and required verification. This is important, for instance, as the journal Cognition recently introduced a progressive and valuable data-sharing policy. While the policy was very successful, there was no required verification. Consequently, only about 80% of the published articles actually complied with the policy (Hardwicke et al., Reference Hardwicke, Mathur, MacDonald, Nilsonne, Banks, Kidwell, Mohr, Clayton, Yoon, Tessler, Lenne, Altman, Long and Frank2018). Thus, the TOP factor is similar to other metrics, in that editors could abuse metrics by posting policies but not enforcing them (this was not the intention of Cognition).

We note that some people may view data-sharing requirements as unnecessary or counterproductive because they could lower study participation and response rates (Gabriel & Wessel, Reference Gabriel and Wessel2013) or thwart the publication of data from consulting firms and test publishers (Jones & Dages, Reference Jones and Dages2013). We point to research from the medical sciences that has successfully dealt with similar issues. For a while now, despite apprehensions and resistance from researchers and the pharmaceutical industry, journals in this scientific area have adopted quite stringent transparency standards (Zarin et al., Reference Zarin, Tse, Williams and Carr2016). Maybe our journals could, too. Also, authors could apply for exceptions regarding data transparency and sharing requirements. The reviewing team (action editor and reviewers) or the editorial board could evaluate such applications on a case-by-case basis and, if warranted, grant them. The PLoS journals have such a process (see http://journals.plos.org/plosone/s/submission-guidelines).

Another, possibly complementary approach to the adoption of the TOP standards pertains to standards that editors might hold themselves to. The Editor Ethics 2.0 Code (https://editorethics.uncc.edu/editor-ethics-2-0-code/) has been established by a series of I-O psychology and management scholars to promote healthy editorial practices and discourage unhealthy practices, such as coercive citation practices. There are other approaches to assessing journal quality as well—some better, some worse. For instance, measuring how applicable and actionable published research is could be important for practitioners (Pearce & Huang, Reference Pearce and Huang2012), especially when considering the evidence-based management movement. Indeed, some scholars have long lamented the seemingly widening gap between science and practice (Pearce & Huang, Reference Pearce and Huang2012) and how evidence-based management could help to overcome it (Briner & Rousseau, Reference Briner and Rousseau2011; Rousseau, Reference Rousseau2012). Yet, not much progress seems to have been made. Indeed, today’s “implications for practice” sections seem to be as devoid of actionable practical knowledge as they were a decade ago (Kieser & Leiner, Reference Kieser and Leiner2009; Pearce & Huang, Reference Pearce and Huang2012). Maybe an index that incorporates an assessment of the practical relevance and actionability of published research should be an important aspect of journal quality. A robustness assessment of the evidence underlying such practical implications should also be a part of such a metric or dimension (Kepes et al., Reference Kepes, Bennett and McDaniel2014). That is why it may make sense to incorporate such assessments more directly into the TOP factor.

In closing, we want to highlight that one should not fall prey to the “tyranny of metrics” by blindly following any one particular index or set of indices (Muller, Reference Muller2019). All indices assess something, and we need to understand the underlying construct a metric is representing before we start using it. After all, very few people would confuse the popularity of cable TV news shows, as assessed by the Nielsen TV ratings, with the accuracy or quality of such shows. Why are we committing this fallacy with metrics such as the JIF? We thus call for an open forum to propose and discuss actual measures of journal quality. We feel that this journal and similar ones, the flagship journals of our leading scientific and professional organizations, and conferences may be good places to start this debate. However, ultimately, it is up to our gatekeepers, especially leaders of our organizations, journal editors, and reviewers, to ensure that our quality standards are being upheld.