Across scientific disciplines, the best journals are the most exclusive; they have the highest rejection rates and biggest impact factors (Day, Reference Day2011; Rotton et al., Reference Rotton, Levitt and Foos1993). In industrial-organizational (I-O) psychology, the most prestigious journals set a high bar for manuscripts to make a significant contribution, be theoretically novel, and be rigorous in research design and execution. Highhouse et al. (Reference Highhouse, Zickar and Melick2020) review and compare conventional metrics for journal quality alongside subjective ratings of journal prestige. Recently, an additional set of criteria has been proposed, by the Center for Open Science (COS), to judge the quality of journals: the Transparency and Openness Promotion (TOP) factor (https://topfactor.org/). The TOP factor rates journals on the transparency and openness requirements they place on authors (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and Yarkoni2015). Some criteria include whether journals require that authors make their data, analysis code, and materials available in online repositories and whether journals require the preregistration of studies. These criteria are generally independent of what most researchers would traditionally equate with methodological rigor and making a contribution but pertain more to emphasizing process elements and certain scientific values (https://cos.io/about/news/new-measure-rates-quality-research-journals-policies-promote-transparency-and-reproducibility/). The TOP factor is not meant to replace existing journal quality metrics but instead is meant to provide additional scientifically relevant information. In light of Highhouse et al.’s analysis and concerns about the credibility of science (e.g., Simmons et al., Reference Simmons, Nelson and Simonsohn2011), we wanted to ask whether I-O psychology should also consider openness and transparency practices when ranking our journals. If so, should our highest ranking journals be the most open and transparent?
On its face, openness and transparency sound like a good idea. They are characteristics that are consistent with how most people would think of science. At the same time, there may be reasons why transparency and openness would not be favored (e.g., data ownership, privacy concerns). One way to understand the expected value of the TOP criteria is to consider the analogy of a restaurant, specifically, a restaurant in a world where restaurants do not undergo health and safety checks. In this fictitious world, there is a popular restaurant where it is always busy, and it is difficult to get a reservation. This restaurant is held in high esteem and everyone you know speaks very highly of it. Like everyone else, you think very highly of the restaurant. Would your opinion of the restaurant change if you learned about their food preparation processes and discovered that the staff were never trained on hygiene practices (e.g., hand washing) or if the kitchen staff was simply “encouraged” to check the expiration date on meat before cooking and serving it? Would you still want to eat there even if the popularity of the restaurant did not change? Perhaps you would still eat there but would now be very selective about what you order from the menu.
The TOP factor may be viewed as a health and safety check on research processes: a way to check the process through which science is generated. Of course, not having checks does not mean that standards are not met. As is sometimes said, absence of evidence is not evidence of absence. However, making such information available could be particularly relevant in an applied field like I-O psychology where there are many consumers of research: practitioners and scientists. Indeed, some concerns surrounding the credibility of research was raised by practitioners in Highhouse et al.’s (Reference Highhouse, Zickar and Melick2020) investigation. A metric like the TOP factor may be a way to make certain process information objective and explicit to consumers of research.
More on the TOP factor
The TOP factor is based on the TOP guidelines, which were first established in 2015 as a set of voluntary guidelines with eight dimensions (e.g., data transparency, analytic methods transparency, research materials transparency) that journals could adhere to at various levels (0 to 3) for each dimension (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and Yarkoni2015). Transparency and openness practices appear to be increasing in popularity, with COS reporting that over 1,100 journals or organizations are implementing some level of compliance with TOP guidelines (https://cos.io/top/).
The newly released TOP factor provides ratings for journals on all eight dimensions plus an additional ninth dimension that indicates whether they reward articles that adhere to TOP guidelines with Open Science badges (https://cos.io/top/). An overall score for each journal (maximum of 27) is provided that allows scientists to quickly assess the extent to which journals adhere to TOP guidelines. Recently, the results of evaluating 346 journals according to the TOP factor were released. The results revealed that although some journals receive high scores (27, 23, and 23 for Meta-Psychology, Cortex, and Advances in Methods and Practices in Psychological Science, respectively) many score considerably lower (e.g., 0, 1, 2). In this initial set of 346 journals, some journals that are relevant to I-O psychology were included (e.g., Academy of Management Learning and Education, The Leadership Quarterly, Journal of Vocational Behavior, Journal of Business and Psychology, among others). The highest score among these journals was 16, obtained by Management and Organization Review. However, only one of most prestigious I-O journals, as ranked by reputation, reported in Highhouse et al. (Reference Highhouse, Zickar and Melick2020; as seen in Table 2), was included in these initial TOP factor ratings (Journal of Applied Psychology, which obtained a score of 3).
How are our most prestigious journals doing?
We decided to see how the other most prestigious I-O journals faired on TOP factor ratings by coding them ourselves. We identified the most prestigious journals as the upper quartile (i.e., upper nine journals) presented by Highhouse et al. (Reference Highhouse, Zickar and Melick2020) in Table 2. Of these nine journals, one was already coded by the COS (as mentioned above) and one does not present empirical data (Academy of Management Review), which means that the TOP factor does not apply; this resulted in us coding seven journals. We coded the website of each journal; we targeted the website sections pertaining to manuscript submission requirements for authors and looked for information that could be used to rate the journal on each of the TOP factor dimensions. We used the TOP factor rubric (https://osf.io/t2yu5/) as the basis for our coding decisions. All three authors independently coded the website for the seven journals. We coded each journal based on what was publicly available on their websites. It may be that manuscript submission portals contain nonpublic information (unavailable to authors prior to submission), which would change a journal’s rating. In some cases, journals had policies on their website for dealing with concerns raised about specific articles (i.e., auditing requirements). This information was not coded, because it did not fit with TOP factor requirements for all articles. Following independent coding, the three authors met and discussed disagreements in order to obtain consensus TOP factor ratings for each journal.
We found that of the eight most prestigious I-O journals, five obtained a TOP factor total score of 0 out of 27, and only three journals obtained nonzero scores. Specifically, the most prestigious I-O psychology journal, Journal of Applied Psychology, obtained a TOP factor total score of 3 out of 27. Both Organizational Behavior and Human Decision Processes and the Journal of Organizational Behavior obtained a TOP factor total score of 2 out of 27. Our coding data and scripts and website files are available in an online repository (=https://osf.io/fuy83/?view_only=c94f3565fb904966848de6180b3f9b01).
To contextualize our results, we downloaded the TOP factor ratings for the 346 journals coded by the COS at the time of writing. We then created a histogram with our results and those from the COS (see Figure 1). This figure reveals that all of the most prestigious I-O journals that we coded are in the largest category (with scores between 0 and 3), which is best described as including journals that merely encourage open processes or do not have any mention of open process on their website—information that is publicly available to authors and readers of the journal.
Figure 1.
Most prestigious I-O journals and other COS-coded journals.
Concluding thoughts
A new metric for assessing journal quality is now available. The extent to which TOP factor will gain traction alongside h-index or impact factor is yet to be determined. Based on our analysis, at present, openness and transparency do not appear to be primary concerns for the most prestigious I-O journals. However, openness, particularly of data, may in some cases be difficult in applied fields due to company and employee privacy concerns. At the same time, members of the Society of Industrial and Organizational Psychology are recognizing the importance of transparent research practices (e.g., Banks et al., Reference Banks, Field, Oswald, O’Boyle, Landis, Rupp and Rogelberg2019).
Even though journal websites do not provide an abundance of content on these issues, we suspect that many high-quality I-O journals may be implementing TOP practices but may not be fully reporting them on their websites. Because the COS relied solely on webpages, we created our TOP factor score the same way. In some cases, there may be additional process checks posted to online submission portals or journal websites may provide links to outside standards or editorial papers with additional information. Based on journal websites, both the initial coding done by the COS and our own coding suggest that, at most, journals are merely encouraging practices such as preregistration and data sharing. Consequently, the modal and median TOP factor score for prestigious I-O journals was zero.
Interestingly, despite the very different methodologies used in calculating impact factor and the survey administered by Highhouse et al. (Reference Highhouse, Zickar and Melick2020), these two metrics were highly correlated. In other words, although the methodology for collecting journal prestige ratings was quite different from how impact factor is calculated, the prestige rating does not provide a great deal of information beyond impact factor. In contrast, we suspect that TOP factor may be uncorrelated with journal prestige ratings or impact factor and would therefore provide additional, nonredundant information about journal quality. However, given the lack of variability in TOP factor scores (a floor effect) a correlation would not be informative at this time.
In considering this issue, we do not want to suggest that the TOP factor replace existing metrics that focus on impact or relevance for I-O journals. Indeed, it would be quite possible for a journal to be very high on the TOP factor but contain studies that answer very unimportant and irrelevant questions. We do not suggest that TOP requirements replace other journal metrics; instead, they might be a preliminary hurdle to be assessed, prior to assessing relevance.
A limited window for the TOP factor?
Although the TOP factor may provide information that is nonredundant with other journal metrics, such as impact factors or subjective prestige ratings, we can foresee a future where this metric becomes obsolete. For instance, if all journals were to wholeheartedly enforce open and transparent processes, then the TOP factor metric will have no variability on the top end (ceiling effect). As a result, it would no longer provide any unique information to differentiate journals. Perhaps this is a goal: adopt open and transparent policies to the extent that a metric of openness and transparency becomes obsolete.
This pattern has happened before. Currently, psychology journals require authors to check that they have received research ethics clearance for data collection from human participants. This requirement does not distinguish one journal from another because it is standard across all journals. Perhaps the TOP factor also will reach such a point in the future. However, until that time comes, we would urge journals to make the required process practices readily available on their website, and we would urge consumers of I-O psychology research to pay attention to the enforcement these practices when deciding what research to consume.
You have
Access
Across scientific disciplines, the best journals are the most exclusive; they have the highest rejection rates and biggest impact factors (Day, Reference Day2011; Rotton et al., Reference Rotton, Levitt and Foos1993). In industrial-organizational (I-O) psychology, the most prestigious journals set a high bar for manuscripts to make a significant contribution, be theoretically novel, and be rigorous in research design and execution. Highhouse et al. (Reference Highhouse, Zickar and Melick2020) review and compare conventional metrics for journal quality alongside subjective ratings of journal prestige. Recently, an additional set of criteria has been proposed, by the Center for Open Science (COS), to judge the quality of journals: the Transparency and Openness Promotion (TOP) factor (https://topfactor.org/). The TOP factor rates journals on the transparency and openness requirements they place on authors (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and Yarkoni2015). Some criteria include whether journals require that authors make their data, analysis code, and materials available in online repositories and whether journals require the preregistration of studies. These criteria are generally independent of what most researchers would traditionally equate with methodological rigor and making a contribution but pertain more to emphasizing process elements and certain scientific values (https://cos.io/about/news/new-measure-rates-quality-research-journals-policies-promote-transparency-and-reproducibility/). The TOP factor is not meant to replace existing journal quality metrics but instead is meant to provide additional scientifically relevant information. In light of Highhouse et al.’s analysis and concerns about the credibility of science (e.g., Simmons et al., Reference Simmons, Nelson and Simonsohn2011), we wanted to ask whether I-O psychology should also consider openness and transparency practices when ranking our journals. If so, should our highest ranking journals be the most open and transparent?
On its face, openness and transparency sound like a good idea. They are characteristics that are consistent with how most people would think of science. At the same time, there may be reasons why transparency and openness would not be favored (e.g., data ownership, privacy concerns). One way to understand the expected value of the TOP criteria is to consider the analogy of a restaurant, specifically, a restaurant in a world where restaurants do not undergo health and safety checks. In this fictitious world, there is a popular restaurant where it is always busy, and it is difficult to get a reservation. This restaurant is held in high esteem and everyone you know speaks very highly of it. Like everyone else, you think very highly of the restaurant. Would your opinion of the restaurant change if you learned about their food preparation processes and discovered that the staff were never trained on hygiene practices (e.g., hand washing) or if the kitchen staff was simply “encouraged” to check the expiration date on meat before cooking and serving it? Would you still want to eat there even if the popularity of the restaurant did not change? Perhaps you would still eat there but would now be very selective about what you order from the menu.
The TOP factor may be viewed as a health and safety check on research processes: a way to check the process through which science is generated. Of course, not having checks does not mean that standards are not met. As is sometimes said, absence of evidence is not evidence of absence. However, making such information available could be particularly relevant in an applied field like I-O psychology where there are many consumers of research: practitioners and scientists. Indeed, some concerns surrounding the credibility of research was raised by practitioners in Highhouse et al.’s (Reference Highhouse, Zickar and Melick2020) investigation. A metric like the TOP factor may be a way to make certain process information objective and explicit to consumers of research.
More on the TOP factor
The TOP factor is based on the TOP guidelines, which were first established in 2015 as a set of voluntary guidelines with eight dimensions (e.g., data transparency, analytic methods transparency, research materials transparency) that journals could adhere to at various levels (0 to 3) for each dimension (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and Yarkoni2015). Transparency and openness practices appear to be increasing in popularity, with COS reporting that over 1,100 journals or organizations are implementing some level of compliance with TOP guidelines (https://cos.io/top/).
The newly released TOP factor provides ratings for journals on all eight dimensions plus an additional ninth dimension that indicates whether they reward articles that adhere to TOP guidelines with Open Science badges (https://cos.io/top/). An overall score for each journal (maximum of 27) is provided that allows scientists to quickly assess the extent to which journals adhere to TOP guidelines. Recently, the results of evaluating 346 journals according to the TOP factor were released. The results revealed that although some journals receive high scores (27, 23, and 23 for Meta-Psychology, Cortex, and Advances in Methods and Practices in Psychological Science, respectively) many score considerably lower (e.g., 0, 1, 2). In this initial set of 346 journals, some journals that are relevant to I-O psychology were included (e.g., Academy of Management Learning and Education, The Leadership Quarterly, Journal of Vocational Behavior, Journal of Business and Psychology, among others). The highest score among these journals was 16, obtained by Management and Organization Review. However, only one of most prestigious I-O journals, as ranked by reputation, reported in Highhouse et al. (Reference Highhouse, Zickar and Melick2020; as seen in Table 2), was included in these initial TOP factor ratings (Journal of Applied Psychology, which obtained a score of 3).
How are our most prestigious journals doing?
We decided to see how the other most prestigious I-O journals faired on TOP factor ratings by coding them ourselves. We identified the most prestigious journals as the upper quartile (i.e., upper nine journals) presented by Highhouse et al. (Reference Highhouse, Zickar and Melick2020) in Table 2. Of these nine journals, one was already coded by the COS (as mentioned above) and one does not present empirical data (Academy of Management Review), which means that the TOP factor does not apply; this resulted in us coding seven journals. We coded the website of each journal; we targeted the website sections pertaining to manuscript submission requirements for authors and looked for information that could be used to rate the journal on each of the TOP factor dimensions. We used the TOP factor rubric (https://osf.io/t2yu5/) as the basis for our coding decisions. All three authors independently coded the website for the seven journals. We coded each journal based on what was publicly available on their websites. It may be that manuscript submission portals contain nonpublic information (unavailable to authors prior to submission), which would change a journal’s rating. In some cases, journals had policies on their website for dealing with concerns raised about specific articles (i.e., auditing requirements). This information was not coded, because it did not fit with TOP factor requirements for all articles. Following independent coding, the three authors met and discussed disagreements in order to obtain consensus TOP factor ratings for each journal.
We found that of the eight most prestigious I-O journals, five obtained a TOP factor total score of 0 out of 27, and only three journals obtained nonzero scores. Specifically, the most prestigious I-O psychology journal, Journal of Applied Psychology, obtained a TOP factor total score of 3 out of 27. Both Organizational Behavior and Human Decision Processes and the Journal of Organizational Behavior obtained a TOP factor total score of 2 out of 27. Our coding data and scripts and website files are available in an online repository (=https://osf.io/fuy83/?view_only=c94f3565fb904966848de6180b3f9b01).
To contextualize our results, we downloaded the TOP factor ratings for the 346 journals coded by the COS at the time of writing. We then created a histogram with our results and those from the COS (see Figure 1). This figure reveals that all of the most prestigious I-O journals that we coded are in the largest category (with scores between 0 and 3), which is best described as including journals that merely encourage open processes or do not have any mention of open process on their website—information that is publicly available to authors and readers of the journal.
Most prestigious I-O journals and other COS-coded journals.
Concluding thoughts
A new metric for assessing journal quality is now available. The extent to which TOP factor will gain traction alongside h-index or impact factor is yet to be determined. Based on our analysis, at present, openness and transparency do not appear to be primary concerns for the most prestigious I-O journals. However, openness, particularly of data, may in some cases be difficult in applied fields due to company and employee privacy concerns. At the same time, members of the Society of Industrial and Organizational Psychology are recognizing the importance of transparent research practices (e.g., Banks et al., Reference Banks, Field, Oswald, O’Boyle, Landis, Rupp and Rogelberg2019).
Even though journal websites do not provide an abundance of content on these issues, we suspect that many high-quality I-O journals may be implementing TOP practices but may not be fully reporting them on their websites. Because the COS relied solely on webpages, we created our TOP factor score the same way. In some cases, there may be additional process checks posted to online submission portals or journal websites may provide links to outside standards or editorial papers with additional information. Based on journal websites, both the initial coding done by the COS and our own coding suggest that, at most, journals are merely encouraging practices such as preregistration and data sharing. Consequently, the modal and median TOP factor score for prestigious I-O journals was zero.
Interestingly, despite the very different methodologies used in calculating impact factor and the survey administered by Highhouse et al. (Reference Highhouse, Zickar and Melick2020), these two metrics were highly correlated. In other words, although the methodology for collecting journal prestige ratings was quite different from how impact factor is calculated, the prestige rating does not provide a great deal of information beyond impact factor. In contrast, we suspect that TOP factor may be uncorrelated with journal prestige ratings or impact factor and would therefore provide additional, nonredundant information about journal quality. However, given the lack of variability in TOP factor scores (a floor effect) a correlation would not be informative at this time.
In considering this issue, we do not want to suggest that the TOP factor replace existing metrics that focus on impact or relevance for I-O journals. Indeed, it would be quite possible for a journal to be very high on the TOP factor but contain studies that answer very unimportant and irrelevant questions. We do not suggest that TOP requirements replace other journal metrics; instead, they might be a preliminary hurdle to be assessed, prior to assessing relevance.
A limited window for the TOP factor?
Although the TOP factor may provide information that is nonredundant with other journal metrics, such as impact factors or subjective prestige ratings, we can foresee a future where this metric becomes obsolete. For instance, if all journals were to wholeheartedly enforce open and transparent processes, then the TOP factor metric will have no variability on the top end (ceiling effect). As a result, it would no longer provide any unique information to differentiate journals. Perhaps this is a goal: adopt open and transparent policies to the extent that a metric of openness and transparency becomes obsolete.
This pattern has happened before. Currently, psychology journals require authors to check that they have received research ethics clearance for data collection from human participants. This requirement does not distinguish one journal from another because it is standard across all journals. Perhaps the TOP factor also will reach such a point in the future. However, until that time comes, we would urge journals to make the required process practices readily available on their website, and we would urge consumers of I-O psychology research to pay attention to the enforcement these practices when deciding what research to consume.