The focal article (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) provides an exemplary roadmap for improving the practice of robust science in industrial-organizational (I-O) psychology, which includes recommendations for authors such as practicing robust science, even in the absence of reward. However, authors are faced with practical constraints; to succeed in our field, we must publish, at least occasionally, in prestigious journals to secure scholarships, awards, employment, grants, and tenure. We can practice robust science (commonly referred to as “open science” or “reproducible research”) without reward—but only to some extent, if there is no support from our gatekeepers. After all, it is difficult to influence the field without a job and research funding.
I reviewed the author guidelines and editorial policies for a variety of relevant journals in our field for support of robust science. Because this is a commentary, the ratings were provided by the author of this manuscript and should be considered a preliminary review of robust science editorial policies in our field. This review is not an indictment of any specific journal but is intended to provide an overall assessment of how our field's academic journals require the practice of robust science. To conduct this review, I crafted a list of peer-reviewed journals in our field by cross referencing Journal Citation Report's (2017) applied psychology journals with Harzing's Journal Quality List for business journals (Reference Harzing2017) and identified 20 journals that appeared on both lists. This narrowed the review to a list of journals relevant to our field. Therefore, this is not an exhaustive review but one that includes many of the most common and well-regarded journals in our area. Next, I reviewed the journal practices and policies recommended by the Center for Open Science's Transparency and Openness Promotion (TOP) guidelines (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015, p. 1424); I omitted “citation standards” because these are already required by the American Psychological Association (APA) and are ubiquitous in psychology. Finally, I visited each journal's website and reviewed their journal descriptions and submission guidelines, mapping how each journal's guidelines corresponded to the TOP Guidelines.
Each TOP category has four levels, ranging from least open (0) to most open (4), with clear and specific criteria for each level. For example, the criteria for the four levels of data transparency (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015, p. 1424) are as follows: Level 0 criteria are that “journal encourages data sharing—or says nothing”; Level 1 criteria are that “article[s] states whether data are available and, if so, where to access them”; Level 2 criteria are that “Data must be posted to a trusted repository. Exceptions must be identified at article submission”; and Level 3 criteria are that “Data must be posted to a trusted repository and reported analyses will be reproduced independently before publication.” Therefore, including a statement encouraging authors to share their data, or being a member of the Committee on Publication Ethics (COPE) are not, by themselves, sufficient to demonstrate openness and transparency according to the guidelines. Instead, TOP guidelines require that journals go further—posting explicit instructions and requirements for each transparency category. If a journal conforms to TOP guidelines for any given category later in the submission process, but does not explicitly state it on its submission guidelines, it is encouraged to revise its submission guidelines accordingly. See the TOP guidelines for a list of the full criteria (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015). The results of the review are reported in Table 1.
Table 1.
TOP Guideline Levels by Journal
The review uncovered three journals that have made significant strides in improving robust science: Journal of Business Psychology, Journal of Personnel Psychology, and Stress and Health. The first two journals explicitly provide options for registered reports. These journals receive a score of 1—instead of 2 or 3—because they allow registered studies but do not provide badges of recognition for them nor require preregistration. These are specific criteria required by the TOP guidelines (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015). Both journals are also receptive to replication studies. Stress and Health encourages sharing of research materials, especially for randomized controlled studies. Thus, there are some outlets for scholars who seek to engage in best practices for robust science. However, the results of our review suggest that the field, as a whole, has a long way to go. None of the journals requires open sharing of research materials, data, or data syntax, or even a statement of why such materials are unavailable. None of the journals provides badges of recognition for studies that embrace varying levels of open-science standards. The majority of journals do not mention replication studies, or even outright discourage them, despite their importance in the advancement of our science.
These findings raise a conundrum for researchers: Practitioners of robust science are at a disadvantage. The focal article (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) does not acknowledge the potential hurdles in publishing robust science in journals that do not value it. For instance, if the majority of submissions do not pre-register hypotheses or analyses and do not include materials, data, or data syntax, those who engage in robust science only provide extra ammunition for rejection. Reviewers may highlight the failed hypotheses and imperfect theory of preregistered studies, whereas those who do not pre-register are free to engage in HARKing to craft compelling narratives. Reviewers can scrutinize shared survey materials and syntax for errors or artificiality, whereas those who do not share materials may evade detection. Reviewers may comb through shared data and data syntax for problems, providing extra scrutiny to robust science submissions, while giving unshared data and data syntax the benefit of the doubt—it is difficult for reviewers to criticize what they do not have. For prestigious journals, with rejection rates exceeding 90%, practicing robust science will not only be unrewarded, it will likely be punished until editorial polices are revised. Robust science researchers will be akin to online daters who draw attention to their flaws—overlooked and neglected—because their competition strategically minimizes its flaws (Toma, Hancock, & Ellison, Reference Toma, Hancock and Ellison2008).
I make three practical suggestions to the Robust and Reliable Science task force to help improve the practice of open science. The first suggestion is to advocate that the Society of Industrial and Organizational Psychology lead by example—create an online-only journal that conforms to all of the TOP “Level 1” (or higher) guidelines. This should include mandatory sharing of all materials, data, and data syntax, and require in-text explanations when such information is not provided (e.g., proprietary tests, organizational data). To help preserve organizational resources, these materials can be housed, for free, at the Open Science Framework (http://osf.io). This new journal should also include the option to preregister study plans and data analyses, and encourage replication studies as short notes. This new journal should provide badges of recognition to articles for their level of commitment to open science, such as shared materials, shared data, or preregistration (see Psychological Science, for an example).
The second suggestion is to leverage learning theory principals to encourage journal editors to improve editorial practices through public acknowledgments. The task force should recommend the creation of a robust science award for editors who make significant improvements in robust science editorial policies, and the award must be well communicated, so that researchers are made aware of these advancements. The third and related recommendation is that the task force compile and distribute an annual robust science rating of all journals in our space. Its review should go further than the one presented here—having dozens of trained expert raters review all journals in our area. Such lists would publically acknowledge journals that practice robust science and boost their profiles. The implementation of these three practical suggestions may provide a significant catalyst for the practice of robust and reliable science in industrial and organizational psychology. It is critical that we act quickly—other branches of psychology have made significant progress encouraging robust science—and we risk looking outdated and less credible as a science if we lag behind.
The focal article (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) provides an exemplary roadmap for improving the practice of robust science in industrial-organizational (I-O) psychology, which includes recommendations for authors such as practicing robust science, even in the absence of reward. However, authors are faced with practical constraints; to succeed in our field, we must publish, at least occasionally, in prestigious journals to secure scholarships, awards, employment, grants, and tenure. We can practice robust science (commonly referred to as “open science” or “reproducible research”) without reward—but only to some extent, if there is no support from our gatekeepers. After all, it is difficult to influence the field without a job and research funding.
I reviewed the author guidelines and editorial policies for a variety of relevant journals in our field for support of robust science. Because this is a commentary, the ratings were provided by the author of this manuscript and should be considered a preliminary review of robust science editorial policies in our field. This review is not an indictment of any specific journal but is intended to provide an overall assessment of how our field's academic journals require the practice of robust science. To conduct this review, I crafted a list of peer-reviewed journals in our field by cross referencing Journal Citation Report's (2017) applied psychology journals with Harzing's Journal Quality List for business journals (Reference Harzing2017) and identified 20 journals that appeared on both lists. This narrowed the review to a list of journals relevant to our field. Therefore, this is not an exhaustive review but one that includes many of the most common and well-regarded journals in our area. Next, I reviewed the journal practices and policies recommended by the Center for Open Science's Transparency and Openness Promotion (TOP) guidelines (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015, p. 1424); I omitted “citation standards” because these are already required by the American Psychological Association (APA) and are ubiquitous in psychology. Finally, I visited each journal's website and reviewed their journal descriptions and submission guidelines, mapping how each journal's guidelines corresponded to the TOP Guidelines.
Each TOP category has four levels, ranging from least open (0) to most open (4), with clear and specific criteria for each level. For example, the criteria for the four levels of data transparency (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015, p. 1424) are as follows: Level 0 criteria are that “journal encourages data sharing—or says nothing”; Level 1 criteria are that “article[s] states whether data are available and, if so, where to access them”; Level 2 criteria are that “Data must be posted to a trusted repository. Exceptions must be identified at article submission”; and Level 3 criteria are that “Data must be posted to a trusted repository and reported analyses will be reproduced independently before publication.” Therefore, including a statement encouraging authors to share their data, or being a member of the Committee on Publication Ethics (COPE) are not, by themselves, sufficient to demonstrate openness and transparency according to the guidelines. Instead, TOP guidelines require that journals go further—posting explicit instructions and requirements for each transparency category. If a journal conforms to TOP guidelines for any given category later in the submission process, but does not explicitly state it on its submission guidelines, it is encouraged to revise its submission guidelines accordingly. See the TOP guidelines for a list of the full criteria (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015). The results of the review are reported in Table 1.
TOP Guideline Levels by Journal
Note. This review is based on journal editorial policies on or before September 18, 2017. Each journal was assessed using the specific criteria outlined in the journal practices and policies recommended by the Center for Open Science's Transparency and Openness Promotion guidelines (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015, p. 1424). The category for citation standards is not included because these are already required by APA.
a Accepts international replications.
b Accepts replication studies (usually as short notes) and sends them for blind review.
c Author guidelines explicitly encourage linking of data, but it is not required.
The review uncovered three journals that have made significant strides in improving robust science: Journal of Business Psychology, Journal of Personnel Psychology, and Stress and Health. The first two journals explicitly provide options for registered reports. These journals receive a score of 1—instead of 2 or 3—because they allow registered studies but do not provide badges of recognition for them nor require preregistration. These are specific criteria required by the TOP guidelines (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015). Both journals are also receptive to replication studies. Stress and Health encourages sharing of research materials, especially for randomized controlled studies. Thus, there are some outlets for scholars who seek to engage in best practices for robust science. However, the results of our review suggest that the field, as a whole, has a long way to go. None of the journals requires open sharing of research materials, data, or data syntax, or even a statement of why such materials are unavailable. None of the journals provides badges of recognition for studies that embrace varying levels of open-science standards. The majority of journals do not mention replication studies, or even outright discourage them, despite their importance in the advancement of our science.
These findings raise a conundrum for researchers: Practitioners of robust science are at a disadvantage. The focal article (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) does not acknowledge the potential hurdles in publishing robust science in journals that do not value it. For instance, if the majority of submissions do not pre-register hypotheses or analyses and do not include materials, data, or data syntax, those who engage in robust science only provide extra ammunition for rejection. Reviewers may highlight the failed hypotheses and imperfect theory of preregistered studies, whereas those who do not pre-register are free to engage in HARKing to craft compelling narratives. Reviewers can scrutinize shared survey materials and syntax for errors or artificiality, whereas those who do not share materials may evade detection. Reviewers may comb through shared data and data syntax for problems, providing extra scrutiny to robust science submissions, while giving unshared data and data syntax the benefit of the doubt—it is difficult for reviewers to criticize what they do not have. For prestigious journals, with rejection rates exceeding 90%, practicing robust science will not only be unrewarded, it will likely be punished until editorial polices are revised. Robust science researchers will be akin to online daters who draw attention to their flaws—overlooked and neglected—because their competition strategically minimizes its flaws (Toma, Hancock, & Ellison, Reference Toma, Hancock and Ellison2008).
I make three practical suggestions to the Robust and Reliable Science task force to help improve the practice of open science. The first suggestion is to advocate that the Society of Industrial and Organizational Psychology lead by example—create an online-only journal that conforms to all of the TOP “Level 1” (or higher) guidelines. This should include mandatory sharing of all materials, data, and data syntax, and require in-text explanations when such information is not provided (e.g., proprietary tests, organizational data). To help preserve organizational resources, these materials can be housed, for free, at the Open Science Framework (http://osf.io). This new journal should also include the option to preregister study plans and data analyses, and encourage replication studies as short notes. This new journal should provide badges of recognition to articles for their level of commitment to open science, such as shared materials, shared data, or preregistration (see Psychological Science, for an example).
The second suggestion is to leverage learning theory principals to encourage journal editors to improve editorial practices through public acknowledgments. The task force should recommend the creation of a robust science award for editors who make significant improvements in robust science editorial policies, and the award must be well communicated, so that researchers are made aware of these advancements. The third and related recommendation is that the task force compile and distribute an annual robust science rating of all journals in our space. Its review should go further than the one presented here—having dozens of trained expert raters review all journals in our area. Such lists would publically acknowledge journals that practice robust science and boost their profiles. The implementation of these three practical suggestions may provide a significant catalyst for the practice of robust and reliable science in industrial and organizational psychology. It is critical that we act quickly—other branches of psychology have made significant progress encouraging robust science—and we risk looking outdated and less credible as a science if we lag behind.